Jul 02 16:56:52.694007 ip-10-0-134-244 systemd[1]: Starting Kubernetes Kubelet... Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.090607 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095492 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095504 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095507 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095511 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095514 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095518 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095521 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 16:56:53.157122 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095524 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095527 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095529 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095532 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095535 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095538 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095541 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095543 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095546 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095549 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095558 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095561 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095564 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095567 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095570 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095572 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095575 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095578 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095582 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095586 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 16:56:53.157995 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095589 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095592 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095595 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095598 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095601 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095604 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095606 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095610 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095612 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095616 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095618 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095621 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095623 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095626 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095629 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095632 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095635 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095638 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095641 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095644 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 16:56:53.158559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095648 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095651 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095654 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095656 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095659 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095661 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095664 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095668 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095670 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095673 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095676 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095678 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095681 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095683 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095686 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095689 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095692 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095695 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095697 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095700 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 16:56:53.159250 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095702 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095705 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095708 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095711 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095714 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095717 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095719 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095722 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095737 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095740 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095742 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095745 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095748 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095751 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095754 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095756 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095759 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095762 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095764 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 16:56:53.159968 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095767 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095770 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095773 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095778 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095781 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095784 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095788 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095792 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095796 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095799 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095802 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095805 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.095808 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097156 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097163 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097166 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097169 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097171 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097175 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 02 16:56:53.160791 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097177 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097180 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097183 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097186 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097189 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097192 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097195 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097198 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097200 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097203 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097205 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097208 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097211 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097214 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097216 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097219 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097222 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097226 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097230 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 16:56:53.161391 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097233 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097236 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097238 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097241 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097244 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097247 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097253 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097257 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097259 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097262 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097265 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097267 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097270 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097274 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097276 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097280 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097284 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097287 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097289 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 16:56:53.161946 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097292 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097295 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097298 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097301 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097303 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097306 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097308 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097312 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097314 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097317 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097320 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097322 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097325 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097327 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097330 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097332 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097335 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097337 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097344 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097348 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 16:56:53.162786 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097351 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 16:56:53.162767 ip-10-0-134-244 systemd[1]: Started Kubernetes Kubelet. Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097354 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097357 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097359 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097362 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097364 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097368 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097371 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097374 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097376 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097379 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097381 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097384 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097386 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097389 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097391 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097394 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097396 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097399 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097401 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 16:56:53.163671 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097404 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097406 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097409 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097411 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097414 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097417 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097420 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097424 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097427 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097430 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097433 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097437 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097439 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097442 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097444 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097835 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097840 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097843 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097845 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 16:56:53.164704 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097849 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097852 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097855 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097857 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097860 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097863 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097866 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097868 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097871 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097873 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097876 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097879 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097881 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097884 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097886 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097889 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097894 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097896 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097899 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097901 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 16:56:53.165559 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097904 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097906 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097909 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097911 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097915 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097918 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097921 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097923 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097926 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097928 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097931 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097933 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097936 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097939 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097942 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097944 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097947 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097950 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097952 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097954 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 16:56:53.166561 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097957 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097960 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097962 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097965 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097967 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097970 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097972 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097975 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097983 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097987 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097990 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097993 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097996 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.097999 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098003 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098008 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098013 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098015 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098018 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 16:56:53.167173 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098021 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098023 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098025 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098028 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098031 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098033 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098036 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098047 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098050 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098052 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098055 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098057 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098060 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098062 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098065 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098067 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098070 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098072 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098075 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098077 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 16:56:53.167997 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098080 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098083 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098086 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098089 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098093 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098096 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098099 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098103 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098105 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098110 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098112 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098115 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098117 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098120 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098122 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098125 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098193 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098204 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098211 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098215 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098222 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 16:56:53.168600 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098225 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098229 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098233 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098236 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098240 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098243 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098246 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098249 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098252 2582 flags.go:64] FLAG: --cgroup-root="" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098255 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098258 2582 flags.go:64] FLAG: --client-ca-file="" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098261 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098264 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098268 2582 flags.go:64] FLAG: --cluster-domain="" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098271 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098275 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098278 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098281 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098284 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098288 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098292 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098295 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098298 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098301 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 16:56:53.169472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098305 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098308 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098311 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098314 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098317 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098320 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098323 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098326 2582 flags.go:64] FLAG: --enable-server="true" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098329 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098334 2582 flags.go:64] FLAG: --event-burst="100" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098337 2582 flags.go:64] FLAG: --event-qps="50" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098340 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098343 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098346 2582 flags.go:64] FLAG: --eviction-hard="" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098350 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098353 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098355 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098359 2582 flags.go:64] FLAG: --eviction-soft="" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098362 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098365 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098369 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098372 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098375 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098377 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098380 2582 flags.go:64] FLAG: --feature-gates="" Jul 02 16:56:53.170426 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098385 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098387 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098391 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098395 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098398 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098401 2582 flags.go:64] FLAG: --help="false" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098404 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098407 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098410 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098413 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098416 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098420 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098423 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098425 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098428 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098431 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098434 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098438 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098441 2582 flags.go:64] FLAG: --kube-reserved="" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098443 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098446 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098449 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098452 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098455 2582 flags.go:64] FLAG: --lock-file="" Jul 02 16:56:53.171293 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098457 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098460 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098463 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098470 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098473 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098476 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098479 2582 flags.go:64] FLAG: --logging-format="text" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098481 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098484 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098487 2582 flags.go:64] FLAG: --manifest-url="" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098490 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098496 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098499 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098502 2582 flags.go:64] FLAG: --max-pods="110" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098505 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098508 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098511 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098514 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098517 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098520 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098523 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098529 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098532 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098535 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 16:56:53.172288 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098538 2582 flags.go:64] FLAG: --pod-cidr="" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098541 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098547 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098549 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098552 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098555 2582 flags.go:64] FLAG: --port="10250" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098558 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098561 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00969421d67e57d44" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098564 2582 flags.go:64] FLAG: --qos-reserved="" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098567 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098570 2582 flags.go:64] FLAG: --register-node="true" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098574 2582 flags.go:64] FLAG: --register-with-taints="" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098578 2582 flags.go:64] FLAG: --registry-burst="10" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098580 2582 flags.go:64] FLAG: --registry-qps="5" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098583 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098586 2582 flags.go:64] FLAG: --reserved-memory="" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098589 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098592 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098595 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098598 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098602 2582 flags.go:64] FLAG: --runonce="false" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098605 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098608 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098610 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098613 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 16:56:53.172978 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098616 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098619 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098622 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098625 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098628 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098631 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098634 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098637 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098640 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098643 2582 flags.go:64] FLAG: --system-cgroups="" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098646 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098651 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098654 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098657 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098660 2582 flags.go:64] FLAG: --tls-min-version="" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098663 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098666 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098668 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098672 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098675 2582 flags.go:64] FLAG: --v="2" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098679 2582 flags.go:64] FLAG: --version="false" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098682 2582 flags.go:64] FLAG: --vmodule="" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098686 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.098689 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098809 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 16:56:53.292901 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098813 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098815 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098820 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098822 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098825 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098828 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098830 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098833 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098835 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098838 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098841 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098843 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098846 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098848 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098851 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098854 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098856 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098860 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098862 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098865 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 16:56:53.312622 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098868 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098870 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098874 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098877 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098880 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098884 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098886 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098889 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098891 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098894 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098896 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098899 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098902 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098904 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098908 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098911 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098913 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098916 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098918 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098921 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 16:56:53.313380 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098923 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098926 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098928 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098931 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098933 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098937 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098941 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098944 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098946 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098949 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098952 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098955 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098957 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098960 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098963 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098965 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098968 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098972 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098974 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 16:56:53.314195 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098977 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098980 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098982 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098984 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098987 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098989 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098992 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098996 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.098998 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099001 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099004 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099006 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099009 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099011 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099014 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099016 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099019 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099021 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099024 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099026 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 16:56:53.314819 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099029 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099031 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099035 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099038 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099042 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099044 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099047 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099050 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099052 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099055 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099059 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099062 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099064 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099067 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099070 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099072 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099075 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099078 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 16:56:53.315471 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.099080 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.099613 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.107000 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.107017 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.107035 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.107041 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107109 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107115 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107119 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107122 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107125 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107128 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107131 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107133 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107136 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 16:56:53.316045 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107138 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107141 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107143 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107146 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107148 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107151 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107155 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107159 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107162 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107165 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107168 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107171 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107174 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107177 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107180 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107182 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107185 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107188 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107191 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 16:56:53.316562 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107194 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107202 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107205 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107208 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107210 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107213 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107215 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107218 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107220 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107223 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107225 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107228 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107231 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107234 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107238 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107242 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107245 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107247 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107250 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107252 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 16:56:53.317109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107255 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107257 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107260 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107264 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107267 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107269 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107272 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107274 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107277 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107279 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107282 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107284 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107287 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107289 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107297 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107300 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107303 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107308 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107311 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107314 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 16:56:53.317690 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107316 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107319 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107321 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107324 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107326 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107329 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107331 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107334 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107336 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107339 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107341 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107344 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107348 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107351 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107354 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107358 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107361 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107371 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107376 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 16:56:53.318253 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107379 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107382 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107385 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107387 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107390 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107392 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107395 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107397 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107406 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107409 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107413 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107415 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.107421 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107583 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107588 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 16:56:53.318858 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107591 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107594 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107597 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107600 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107602 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107605 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107608 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107610 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107614 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107618 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107621 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107624 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107626 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107630 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107633 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107635 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107638 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107640 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107643 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 16:56:53.319280 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107646 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107648 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107651 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107653 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107656 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107658 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107661 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107670 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107673 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107678 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107680 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107683 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107685 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107688 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107690 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107693 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107695 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107698 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107701 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107703 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 16:56:53.320046 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107706 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107708 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107711 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107713 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107716 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107719 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107737 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107740 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107743 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107745 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107748 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107751 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107753 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107756 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107758 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107761 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107763 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107766 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107768 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107771 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 16:56:53.320666 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107779 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107783 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107786 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107789 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107792 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107795 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107798 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107800 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107803 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107805 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107808 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107810 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107813 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107815 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107817 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107820 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107822 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107825 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107829 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 16:56:53.321283 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107831 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107834 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107837 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107839 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107843 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107846 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107849 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107852 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107855 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107858 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107861 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107864 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107867 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107869 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107879 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107882 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107884 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107887 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 16:56:53.321889 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:53.107890 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.107895 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.108039 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.109995 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.111301 2582 server.go:1019] "Starting client certificate rotation" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.111400 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.111438 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.114483 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.116504 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.130179 2582 log.go:25] "Validated CRI v1 runtime API" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.132136 2582 log.go:25] "Validated CRI v1 image API" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.132892 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 16:56:53.322486 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.135257 2582 fs.go:135] Filesystem UUIDs: map[5a8ad4e5-814f-4e2e-9770-3b0adcf81c7e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a927dedb-e9b9-4903-a169-a59ba57f9f93:/dev/nvme0n1p4] Jul 02 16:56:53.322885 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.135272 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 16:56:53.322885 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.140117 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.141017 2582 manager.go:216] Machine: {Timestamp:2026-07-02 16:56:53.139671935 +0000 UTC m=+0.328988957 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100157 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec286d48144be652bc0f014bd7092c4f SystemUUID:ec286d48-144b-e652-bc0f-014bd7092c4f BootID:54e4eed4-f08a-4696-843b-43bc49930de4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ba:f9:40:5a:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ba:f9:40:5a:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:95:54:8c:45:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.141117 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.141191 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.144278 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.144300 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.145477 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.145487 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.145495 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.146134 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.147153 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.150494 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.150609 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.150637 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.150653 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.150667 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.150713 2582 kubelet.go:398] "Adding apiserver pod source" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.150738 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 16:56:53.323544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.151148 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.151162 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.154683 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.156595 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.156623 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.156787 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.156800 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.156805 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.156811 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.156819 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.156824 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.157367 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.157373 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.157379 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.157384 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.157392 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.157400 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.158753 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.158760 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.161943 2582 server.go:1268] "Started kubelet" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.162046 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.162062 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.162112 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.162557 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.162582 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-244.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.162644 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.163285 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.169769 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.170885 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.171096 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-244.ec2.internal.18be876ef7fbb8c3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-244.ec2.internal,UID:ip-10-0-134-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-244.ec2.internal,},FirstTimestamp:2026-07-02 16:56:53.161916611 +0000 UTC m=+0.351233632,LastTimestamp:2026-07-02 16:56:53.161916611 +0000 UTC m=+0.351233632,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-244.ec2.internal,}" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.174578 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.174667 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.174583 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175081 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175118 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175134 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175209 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175221 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175606 2582 factory.go:55] Registering systemd factory Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175622 2582 factory.go:223] Registration of the systemd container factory successfully Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175816 2582 factory.go:153] Registering CRI-O factory Jul 02 16:56:53.474633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175829 2582 factory.go:223] Registration of the crio container factory successfully Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175869 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175888 2582 factory.go:103] Registering Raw factory Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.175897 2582 manager.go:1196] Started watching for new ooms in manager Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.176472 2582 manager.go:349] Starting recovery of all containers Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.179496 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.179623 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.181755 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58pxz" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.188023 2582 manager.go:354] Recovery completed Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.189971 2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.190286 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58pxz" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.193947 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.196024 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientMemory" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.196050 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.196059 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientPID" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.196580 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.196590 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.196590 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.196621 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.197279 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientMemory" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.197301 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.197311 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientPID" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.198128 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-244.ec2.internal.18be876efa0457ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-244.ec2.internal,UID:ip-10-0-134-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-244.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-244.ec2.internal,},FirstTimestamp:2026-07-02 16:56:53.196036079 +0000 UTC m=+0.385353104,LastTimestamp:2026-07-02 16:56:53.196036079 +0000 UTC m=+0.385353104,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-244.ec2.internal,}" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.202661 2582 policy_none.go:49] "None policy: Start" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.202673 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.202683 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.203932 2582 policy_none.go:47] "Start" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.246377 2582 manager.go:337] "Starting Device Plugin manager" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.246536 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.246546 2582 server.go:87] "Starting device plugin registration server" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.246815 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.246827 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.246929 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.247029 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.247049 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.247530 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.247591 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.258137 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 16:56:53.477172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.259259 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.259302 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.259329 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.259374 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.261620 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.347841 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.349461 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientMemory" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.349496 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.349509 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientPID" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.349531 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.355933 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.355952 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-244.ec2.internal\": node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.359429 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal"] Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.359496 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.360847 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientMemory" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.360872 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.360881 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientPID" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.362980 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.363154 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.363180 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.363673 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientMemory" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.363676 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientMemory" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.363694 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.363696 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.363707 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientPID" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.363712 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientPID" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.365938 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.365960 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.366577 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientMemory" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.366600 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.366616 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeHasSufficientPID" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.376567 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7324e3503b6cdcf77c0c7de22f620921-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal\" (UID: \"7324e3503b6cdcf77c0c7de22f620921\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.376588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/282f21947ce88ed32d4f969496593dd1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-244.ec2.internal\" (UID: \"282f21947ce88ed32d4f969496593dd1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.376603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7324e3503b6cdcf77c0c7de22f620921-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal\" (UID: \"7324e3503b6cdcf77c0c7de22f620921\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.380136 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:53.478380 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.401531 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-244.ec2.internal\" not found" node="ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.479438 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.407664 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-244.ec2.internal\" not found" node="ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.479438 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.476840 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/282f21947ce88ed32d4f969496593dd1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-244.ec2.internal\" (UID: \"282f21947ce88ed32d4f969496593dd1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.479438 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.476872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7324e3503b6cdcf77c0c7de22f620921-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal\" (UID: \"7324e3503b6cdcf77c0c7de22f620921\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.479438 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.476895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7324e3503b6cdcf77c0c7de22f620921-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal\" (UID: \"7324e3503b6cdcf77c0c7de22f620921\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.479438 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.476959 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7324e3503b6cdcf77c0c7de22f620921-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal\" (UID: \"7324e3503b6cdcf77c0c7de22f620921\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.479438 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.476972 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/282f21947ce88ed32d4f969496593dd1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-244.ec2.internal\" (UID: \"282f21947ce88ed32d4f969496593dd1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.479438 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.477009 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7324e3503b6cdcf77c0c7de22f620921-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal\" (UID: \"7324e3503b6cdcf77c0c7de22f620921\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.679377 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.480629 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:53.679377 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.581638 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:53.683233 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.681902 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:53.836055 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.705824 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.836055 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:53.711092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" Jul 02 16:56:53.836055 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.782549 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.051620 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.883571 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.051620 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:53.983937 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.277377 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:54.085669 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.277377 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.110929 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 16:56:54.277377 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.111095 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 16:56:54.277377 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:54.190599 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.277377 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.202852 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 16:51:54 +0000 UTC" deadline="2027-12-03 09:35:15.279064056 +0000 UTC" Jul 02 16:56:54.277377 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.202885 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12448h38m21.076183498s" Jul 02 16:56:54.421522 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:54.290947 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.421522 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.308984 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 16:56:54.421522 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:54.391659 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.421522 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:54.411110 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282f21947ce88ed32d4f969496593dd1.slice/crio-6c638cc8028aac7cfae4149f5dd34ea5cc5e59e23f268d5ed5b197ac2e7d42ac WatchSource:0}: Error finding container 6c638cc8028aac7cfae4149f5dd34ea5cc5e59e23f268d5ed5b197ac2e7d42ac: Status 404 returned error can't find the container with id 6c638cc8028aac7cfae4149f5dd34ea5cc5e59e23f268d5ed5b197ac2e7d42ac Jul 02 16:56:54.421522 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:54.413136 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7324e3503b6cdcf77c0c7de22f620921.slice/crio-8c1c5698f2436ca4693f44e03d5d9b70354746ee9b80ab5d3678ffff4c30c4d6 WatchSource:0}: Error finding container 8c1c5698f2436ca4693f44e03d5d9b70354746ee9b80ab5d3678ffff4c30c4d6: Status 404 returned error can't find the container with id 8c1c5698f2436ca4693f44e03d5d9b70354746ee9b80ab5d3678ffff4c30c4d6 Jul 02 16:56:54.497441 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.425004 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 16:56:54.497441 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:54.492026 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.678497 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:54.593430 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-244.ec2.internal\" not found" Jul 02 16:56:54.678497 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.610291 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 16:56:54.678497 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.666750 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 16:56:54.678497 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.675050 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" Jul 02 16:56:54.864636 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.685797 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 16:56:54.864636 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.691725 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" Jul 02 16:56:54.864636 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:54.699660 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.155574 2582 apiserver.go:52] "Watching apiserver" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.163938 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.165229 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-2d9jh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w","openshift-cluster-node-tuning-operator/tuned-925x8","openshift-dns/node-resolver-pfmsz","openshift-image-registry/node-ca-bwj5n","openshift-multus/multus-additional-cni-plugins-2zrq4","openshift-multus/multus-cnkgs","openshift-multus/network-metrics-daemon-tr57l","openshift-network-operator/iptables-alerter-xn7sv","kube-system/global-pull-secret-syncer-rn2wj","kube-system/konnectivity-agent-bx746","kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal","openshift-network-diagnostics/network-check-target-g6mlh"] Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.167981 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.170392 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bx746" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.170482 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.170616 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.170878 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.170931 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-85657\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.171071 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.172475 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.173143 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.173848 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b2sp5\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.173883 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.175616 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.176034 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.176347 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.176409 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6xwrv\"" Jul 02 16:56:55.177664 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.176432 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 16:56:55.179345 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.179009 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.180961 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.179854 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 16:56:55.180961 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.180083 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 16:56:55.180961 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.180315 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k52bn\"" Jul 02 16:56:55.183303 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.181356 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 16:56:55.183303 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.181635 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b278m\"" Jul 02 16:56:55.183303 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.181925 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 16:56:55.183303 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.182313 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.183303 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.182337 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.184257 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.185157 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.185450 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p9p7k\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.185708 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.186442 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.186686 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.186762 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.186867 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.187014 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72dnn\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.187310 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.189192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.190340 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.191887 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.192033 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.192166 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.192800 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tsc48\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.193148 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.193223 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.196704 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.198931 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.198992 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.199213 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.199237 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9cqpn\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.199333 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.199619 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.200202 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.200314 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.200206 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.207713 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdbxr" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.216356 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdbxr" Jul 02 16:56:55.241960 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219220 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-cnibin\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-cni-bin\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219288 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjjfp\" (UniqueName: \"kubernetes.io/projected/ed13545a-affc-489d-a843-24e0e2344de6-kube-api-access-hjjfp\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysctl-conf\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219356 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc678e74-bdb7-47dd-8302-5cfc86a9864b-tmp-dir\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysconfig\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219407 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysctl-d\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219429 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-os-release\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-k8s-cni-cncf-io\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-etc-kubernetes\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219505 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-sys\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-lib-modules\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219556 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc678e74-bdb7-47dd-8302-5cfc86a9864b-hosts-file\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-system-cni-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-socket-dir-parent\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-kubelet\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-netns\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.243494 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36ccebec-913b-4330-8a79-43988fae2af6-konnectivity-ca\") pod \"konnectivity-agent-bx746\" (UID: \"36ccebec-913b-4330-8a79-43988fae2af6\") " pod="kube-system/konnectivity-agent-bx746" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219800 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-kubernetes\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57708e87-f539-4125-b007-dd9026657fc5-tmp\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219893 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-run\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-var-lib-kubelet\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-host\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219954 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57708e87-f539-4125-b007-dd9026657fc5-etc-tuned\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-cni-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.219987 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed13545a-affc-489d-a843-24e0e2344de6-cni-binary-copy\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220001 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-hostroot\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-modprobe-d\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-cni-multus\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220051 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed13545a-affc-489d-a843-24e0e2344de6-multus-daemon-config\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-multus-certs\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxjp9\" (UniqueName: \"kubernetes.io/projected/57708e87-f539-4125-b007-dd9026657fc5-kube-api-access-vxjp9\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220126 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8zh\" (UniqueName: \"kubernetes.io/projected/dc678e74-bdb7-47dd-8302-5cfc86a9864b-kube-api-access-dc8zh\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-conf-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.251866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36ccebec-913b-4330-8a79-43988fae2af6-agent-certs\") pod \"konnectivity-agent-bx746\" (UID: \"36ccebec-913b-4330-8a79-43988fae2af6\") " pod="kube-system/konnectivity-agent-bx746" Jul 02 16:56:55.253238 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.220262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-systemd\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.269257 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.269146 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" event={"ID":"7324e3503b6cdcf77c0c7de22f620921","Type":"ContainerStarted","Data":"8c1c5698f2436ca4693f44e03d5d9b70354746ee9b80ab5d3678ffff4c30c4d6"} Jul 02 16:56:55.270980 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.270866 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" event={"ID":"282f21947ce88ed32d4f969496593dd1","Type":"ContainerStarted","Data":"6c638cc8028aac7cfae4149f5dd34ea5cc5e59e23f268d5ed5b197ac2e7d42ac"} Jul 02 16:56:55.276420 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.276348 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 16:56:55.324256 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-cnibin\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysctl-conf\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-os-release\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-sys-fs\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-socket-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysconfig\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-k8s-cni-cncf-io\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-etc-kubernetes\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc678e74-bdb7-47dd-8302-5cfc86a9864b-hosts-file\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-log-socket\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324726 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-system-cni-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.324807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324784 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-kubelet-config\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-node-log\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-cni-netd\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324912 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysconfig\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.324999 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovnkube-config\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325039 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-system-cni-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlrt\" (UniqueName: \"kubernetes.io/projected/78775583-a909-469d-ac1c-8ff36fa51ccc-kube-api-access-2tlrt\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325090 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-netns\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36ccebec-913b-4330-8a79-43988fae2af6-konnectivity-ca\") pod \"konnectivity-agent-bx746\" (UID: \"36ccebec-913b-4330-8a79-43988fae2af6\") " pod="kube-system/konnectivity-agent-bx746" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57708e87-f539-4125-b007-dd9026657fc5-tmp\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmm2\" (UniqueName: \"kubernetes.io/projected/e1e076a9-2f26-4006-ad54-cd2d108d34de-kube-api-access-kpmm2\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325193 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysctl-conf\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325295 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-cnibin\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-device-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-cni-bin\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rhf\" (UniqueName: \"kubernetes.io/projected/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-kube-api-access-49rhf\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:55.325459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-run\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-host\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325410 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc678e74-bdb7-47dd-8302-5cfc86a9864b-hosts-file\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325526 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-host\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325537 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-etc-kubernetes\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325565 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-run\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325579 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-netns\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325580 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-k8s-cni-cncf-io\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-system-cni-dir\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-registration-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325764 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-hostroot\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-cnibin\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325884 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-hostroot\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-systemd-units\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325929 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-slash\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.325975 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36ccebec-913b-4330-8a79-43988fae2af6-konnectivity-ca\") pod \"konnectivity-agent-bx746\" (UID: \"36ccebec-913b-4330-8a79-43988fae2af6\") " pod="kube-system/konnectivity-agent-bx746" Jul 02 16:56:55.326347 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326054 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326094 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326130 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-cni-multus\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326162 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-multus-certs\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326227 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-run-multus-certs\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326269 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-systemd\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326280 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-cni-multus\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326362 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78775583-a909-469d-ac1c-8ff36fa51ccc-iptables-alerter-script\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-conf-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slc57\" (UniqueName: \"kubernetes.io/projected/2fc75340-7358-46c0-882d-daf8b446a05c-kube-api-access-slc57\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326538 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-run-netns\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326563 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-env-overrides\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovnkube-script-lib\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326625 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-cni-bin\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjjfp\" (UniqueName: \"kubernetes.io/projected/ed13545a-affc-489d-a843-24e0e2344de6-kube-api-access-hjjfp\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.327273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc678e74-bdb7-47dd-8302-5cfc86a9864b-tmp-dir\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-kubelet\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-serviceca\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326849 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysctl-d\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-os-release\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.326927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-sys\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327130 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-os-release\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-lib-modules\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327219 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-sys\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-socket-dir-parent\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327266 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-cni-bin\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327340 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-socket-dir-parent\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327346 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-conf-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-kubelet\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327395 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-lib-modules\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327598 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-sysctl-d\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327644 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-var-lib-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.328076 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-kubernetes\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327753 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-ovn\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovn-node-metrics-cert\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327822 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc678e74-bdb7-47dd-8302-5cfc86a9864b-tmp-dir\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78775583-a909-469d-ac1c-8ff36fa51ccc-host-slash\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-var-lib-kubelet\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327937 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-kubernetes\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327978 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57708e87-f539-4125-b007-dd9026657fc5-etc-tuned\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.327989 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-host-var-lib-kubelet\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328038 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-var-lib-kubelet\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328096 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-dbus\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-cni-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed13545a-affc-489d-a843-24e0e2344de6-cni-binary-copy\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328359 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed13545a-affc-489d-a843-24e0e2344de6-multus-cni-dir\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-modprobe-d\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.329039 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed13545a-affc-489d-a843-24e0e2344de6-multus-daemon-config\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxjp9\" (UniqueName: \"kubernetes.io/projected/57708e87-f539-4125-b007-dd9026657fc5-kube-api-access-vxjp9\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328553 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-modprobe-d\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8zh\" (UniqueName: \"kubernetes.io/projected/dc678e74-bdb7-47dd-8302-5cfc86a9864b-kube-api-access-dc8zh\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328600 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328953 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed13545a-affc-489d-a843-24e0e2344de6-cni-binary-copy\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.328966 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-etc-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.329019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvsm2\" (UniqueName: \"kubernetes.io/projected/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-kube-api-access-mvsm2\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.329072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-host\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.329146 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv7zj\" (UniqueName: \"kubernetes.io/projected/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-kube-api-access-sv7zj\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.329176 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed13545a-affc-489d-a843-24e0e2344de6-multus-daemon-config\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.329193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36ccebec-913b-4330-8a79-43988fae2af6-agent-certs\") pod \"konnectivity-agent-bx746\" (UID: \"36ccebec-913b-4330-8a79-43988fae2af6\") " pod="kube-system/konnectivity-agent-bx746" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.329239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-systemd\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.329918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.329362 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57708e87-f539-4125-b007-dd9026657fc5-etc-systemd\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.333631 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.333583 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57708e87-f539-4125-b007-dd9026657fc5-etc-tuned\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.333759 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.333691 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57708e87-f539-4125-b007-dd9026657fc5-tmp\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.338390 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.338327 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8zh\" (UniqueName: \"kubernetes.io/projected/dc678e74-bdb7-47dd-8302-5cfc86a9864b-kube-api-access-dc8zh\") pod \"node-resolver-pfmsz\" (UID: \"dc678e74-bdb7-47dd-8302-5cfc86a9864b\") " pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.339911 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.339685 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxjp9\" (UniqueName: \"kubernetes.io/projected/57708e87-f539-4125-b007-dd9026657fc5-kube-api-access-vxjp9\") pod \"tuned-925x8\" (UID: \"57708e87-f539-4125-b007-dd9026657fc5\") " pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.340020 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.339907 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36ccebec-913b-4330-8a79-43988fae2af6-agent-certs\") pod \"konnectivity-agent-bx746\" (UID: \"36ccebec-913b-4330-8a79-43988fae2af6\") " pod="kube-system/konnectivity-agent-bx746" Jul 02 16:56:55.340956 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.340908 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjjfp\" (UniqueName: \"kubernetes.io/projected/ed13545a-affc-489d-a843-24e0e2344de6-kube-api-access-hjjfp\") pod \"multus-cnkgs\" (UID: \"ed13545a-affc-489d-a843-24e0e2344de6\") " pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.429611 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.429395 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 16:56:55.429858 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.429818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-os-release\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.429997 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.429863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-sys-fs\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.429997 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.429895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.429997 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.429924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-socket-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.429997 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.429953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.429997 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.429978 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-log-socket\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-kubelet-config\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-node-log\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-cni-netd\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovnkube-config\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlrt\" (UniqueName: \"kubernetes.io/projected/78775583-a909-469d-ac1c-8ff36fa51ccc-kube-api-access-2tlrt\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmm2\" (UniqueName: \"kubernetes.io/projected/e1e076a9-2f26-4006-ad54-cd2d108d34de-kube-api-access-kpmm2\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-device-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-cni-bin\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rhf\" (UniqueName: \"kubernetes.io/projected/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-kube-api-access-49rhf\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430290 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-system-cni-dir\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.430317 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-registration-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-cnibin\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-systemd-units\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-slash\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-systemd\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78775583-a909-469d-ac1c-8ff36fa51ccc-iptables-alerter-script\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slc57\" (UniqueName: \"kubernetes.io/projected/2fc75340-7358-46c0-882d-daf8b446a05c-kube-api-access-slc57\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430670 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-run-netns\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-env-overrides\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovnkube-script-lib\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-kubelet\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.431023 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-serviceca\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-var-lib-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-ovn\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430948 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovn-node-metrics-cert\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.430974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78775583-a909-469d-ac1c-8ff36fa51ccc-host-slash\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431028 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-dbus\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-etc-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvsm2\" (UniqueName: \"kubernetes.io/projected/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-kube-api-access-mvsm2\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-host\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv7zj\" (UniqueName: \"kubernetes.io/projected/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-kube-api-access-sv7zj\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431614 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-os-release\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.432013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.431686 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-sys-fs\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.432865 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.432313 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.432865 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.432442 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-socket-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.432865 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.432506 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-systemd\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433042 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.432950 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.433042 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.432958 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-ovn\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433042 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433021 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-run-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433042 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433036 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.433255 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433074 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-log-socket\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433255 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433126 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-kubelet-config\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.433255 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433166 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-node-log\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433255 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433219 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433483 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433259 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-cni-netd\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433483 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433334 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-run-netns\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433819 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433775 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-env-overrides\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433900 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433849 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovnkube-config\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.433900 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.433885 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78775583-a909-469d-ac1c-8ff36fa51ccc-host-slash\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.434004 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.433921 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 16:56:55.434070 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.434009 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret podName:8ffdd0d1-7f10-427b-8ed5-5b877f73840e nodeName:}" failed. No retries permitted until 2026-07-02 16:56:55.933968328 +0000 UTC m=+3.123285339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret") pod "global-pull-secret-syncer-rn2wj" (UID: "8ffdd0d1-7f10-427b-8ed5-5b877f73840e") : object "kube-system"/"original-pull-secret" not registered Jul 02 16:56:55.434070 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434010 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.434260 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434163 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.434327 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434261 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-serviceca\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.434327 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434297 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-device-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.434327 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434311 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-var-lib-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.434327 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434300 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-dbus\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.434543 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434344 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-cni-bin\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.434543 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434363 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-etc-openvswitch\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.434543 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434425 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-host\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.434543 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434500 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78775583-a909-469d-ac1c-8ff36fa51ccc-iptables-alerter-script\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.434543 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434506 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-kubelet\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.434823 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434579 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-system-cni-dir\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.434823 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434584 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-systemd-units\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.434823 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434634 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.434823 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434672 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-host-slash\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.434823 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434719 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fc75340-7358-46c0-882d-daf8b446a05c-registration-dir\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.434823 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434722 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1e076a9-2f26-4006-ad54-cd2d108d34de-cnibin\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.434823 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434791 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovnkube-script-lib\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.434823 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.434824 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:56:55.435238 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.434880 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs podName:0535e4ec-90d9-47eb-9cd6-9f1e9225a103 nodeName:}" failed. No retries permitted until 2026-07-02 16:56:55.934863873 +0000 UTC m=+3.124180889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs") pod "network-metrics-daemon-tr57l" (UID: "0535e4ec-90d9-47eb-9cd6-9f1e9225a103") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:56:55.435238 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.434897 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1e076a9-2f26-4006-ad54-cd2d108d34de-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.437835 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.437795 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-ovn-node-metrics-cert\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.441255 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.440974 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv7zj\" (UniqueName: \"kubernetes.io/projected/ab2a2297-66d2-4a77-8e4c-b07cbe46725c-kube-api-access-sv7zj\") pod \"node-ca-bwj5n\" (UID: \"ab2a2297-66d2-4a77-8e4c-b07cbe46725c\") " pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.441255 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.441088 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 16:56:55.441255 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.441112 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 16:56:55.441255 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.441125 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9bps9 for pod openshift-network-diagnostics/network-check-target-g6mlh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:56:55.441255 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.441219 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9 podName:5055abe2-4719-4230-91ac-e32ddeab3e0d nodeName:}" failed. No retries permitted until 2026-07-02 16:56:55.941197528 +0000 UTC m=+3.130514541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9bps9" (UniqueName: "kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9") pod "network-check-target-g6mlh" (UID: "5055abe2-4719-4230-91ac-e32ddeab3e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:56:55.444274 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.444211 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slc57\" (UniqueName: \"kubernetes.io/projected/2fc75340-7358-46c0-882d-daf8b446a05c-kube-api-access-slc57\") pod \"aws-ebs-csi-driver-node-xxq6w\" (UID: \"2fc75340-7358-46c0-882d-daf8b446a05c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.446003 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.445965 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlrt\" (UniqueName: \"kubernetes.io/projected/78775583-a909-469d-ac1c-8ff36fa51ccc-kube-api-access-2tlrt\") pod \"iptables-alerter-xn7sv\" (UID: \"78775583-a909-469d-ac1c-8ff36fa51ccc\") " pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.446118 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.446011 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rhf\" (UniqueName: \"kubernetes.io/projected/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-kube-api-access-49rhf\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:55.446637 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.446617 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvsm2\" (UniqueName: \"kubernetes.io/projected/51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd-kube-api-access-mvsm2\") pod \"ovnkube-node-2d9jh\" (UID: \"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.447542 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.447493 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmm2\" (UniqueName: \"kubernetes.io/projected/e1e076a9-2f26-4006-ad54-cd2d108d34de-kube-api-access-kpmm2\") pod \"multus-additional-cni-plugins-2zrq4\" (UID: \"e1e076a9-2f26-4006-ad54-cd2d108d34de\") " pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.493380 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.493260 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cnkgs" Jul 02 16:56:55.502589 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.502560 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bx746" Jul 02 16:56:55.512689 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.512667 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-925x8" Jul 02 16:56:55.518865 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.518842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pfmsz" Jul 02 16:56:55.527906 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.527885 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bwj5n" Jul 02 16:56:55.536487 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.536465 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" Jul 02 16:56:55.608855 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.608816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xn7sv" Jul 02 16:56:55.625322 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.625291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" Jul 02 16:56:55.636566 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.636540 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:56:55.935669 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.935595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:55.935847 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:55.935668 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:55.935847 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.935819 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 16:56:55.935847 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.935845 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:56:55.935990 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.935911 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs podName:0535e4ec-90d9-47eb-9cd6-9f1e9225a103 nodeName:}" failed. No retries permitted until 2026-07-02 16:56:56.935892372 +0000 UTC m=+4.125209398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs") pod "network-metrics-daemon-tr57l" (UID: "0535e4ec-90d9-47eb-9cd6-9f1e9225a103") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:56:55.935990 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:55.935938 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret podName:8ffdd0d1-7f10-427b-8ed5-5b877f73840e nodeName:}" failed. No retries permitted until 2026-07-02 16:56:56.935918336 +0000 UTC m=+4.125235353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret") pod "global-pull-secret-syncer-rn2wj" (UID: "8ffdd0d1-7f10-427b-8ed5-5b877f73840e") : object "kube-system"/"original-pull-secret" not registered Jul 02 16:56:56.036823 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:56.036779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:56:56.037025 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:56.036947 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 16:56:56.037025 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:56.036978 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 16:56:56.037025 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:56.036991 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9bps9 for pod openshift-network-diagnostics/network-check-target-g6mlh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:56:56.037188 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:56.037069 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9 podName:5055abe2-4719-4230-91ac-e32ddeab3e0d nodeName:}" failed. No retries permitted until 2026-07-02 16:56:57.037047215 +0000 UTC m=+4.226364227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bps9" (UniqueName: "kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9") pod "network-check-target-g6mlh" (UID: "5055abe2-4719-4230-91ac-e32ddeab3e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:56:56.220064 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:56.219971 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 16:51:55 +0000 UTC" deadline="2028-02-03 15:39:55.422796725 +0000 UTC" Jul 02 16:56:56.220064 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:56.220010 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13942h42m59.202790594s" Jul 02 16:56:56.530521 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:56.530473 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78775583_a909_469d_ac1c_8ff36fa51ccc.slice/crio-b9ab62ee7502a8b583d79ce9029ee9b432fd956b8b76cb054748ac75a923bf39 WatchSource:0}: Error finding container b9ab62ee7502a8b583d79ce9029ee9b432fd956b8b76cb054748ac75a923bf39: Status 404 returned error can't find the container with id b9ab62ee7502a8b583d79ce9029ee9b432fd956b8b76cb054748ac75a923bf39 Jul 02 16:56:56.532285 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:56.532256 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded13545a_affc_489d_a843_24e0e2344de6.slice/crio-a8b0ee07231f13dba082624d186490a224195e0e196f0e9a398ca44d4a96d54a WatchSource:0}: Error finding container a8b0ee07231f13dba082624d186490a224195e0e196f0e9a398ca44d4a96d54a: Status 404 returned error can't find the container with id a8b0ee07231f13dba082624d186490a224195e0e196f0e9a398ca44d4a96d54a Jul 02 16:56:56.536534 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:56.536510 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc75340_7358_46c0_882d_daf8b446a05c.slice/crio-8dd0a01ae970fbaca2d3981c210c02086a4c0a667290df8aa1473c23b90e2697 WatchSource:0}: Error finding container 8dd0a01ae970fbaca2d3981c210c02086a4c0a667290df8aa1473c23b90e2697: Status 404 returned error can't find the container with id 8dd0a01ae970fbaca2d3981c210c02086a4c0a667290df8aa1473c23b90e2697 Jul 02 16:56:56.537172 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:56.537152 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc678e74_bdb7_47dd_8302_5cfc86a9864b.slice/crio-2a30c514019c29f6854dbced817a816a04d84634e5aa9dc601973f1c04289b5e WatchSource:0}: Error finding container 2a30c514019c29f6854dbced817a816a04d84634e5aa9dc601973f1c04289b5e: Status 404 returned error can't find the container with id 2a30c514019c29f6854dbced817a816a04d84634e5aa9dc601973f1c04289b5e Jul 02 16:56:56.538772 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:56.538752 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e076a9_2f26_4006_ad54_cd2d108d34de.slice/crio-3aaa7776f9a6ec3696391f8ed0a1a1991e3be381edf7eabd6cd3b6115c8e037f WatchSource:0}: Error finding container 3aaa7776f9a6ec3696391f8ed0a1a1991e3be381edf7eabd6cd3b6115c8e037f: Status 404 returned error can't find the container with id 3aaa7776f9a6ec3696391f8ed0a1a1991e3be381edf7eabd6cd3b6115c8e037f Jul 02 16:56:56.539574 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:56:56.539507 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57708e87_f539_4125_b007_dd9026657fc5.slice/crio-3b4e42100d91cd7c053da12038308250cb23528c293e5cc770cb58c2f27774e7 WatchSource:0}: Error finding container 3b4e42100d91cd7c053da12038308250cb23528c293e5cc770cb58c2f27774e7: Status 404 returned error can't find the container with id 3b4e42100d91cd7c053da12038308250cb23528c293e5cc770cb58c2f27774e7 Jul 02 16:56:56.944187 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:56.943836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:56.944187 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:56.944161 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:56.944723 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:56.944053 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:56:56.944723 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:56.944398 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs podName:0535e4ec-90d9-47eb-9cd6-9f1e9225a103 nodeName:}" failed. No retries permitted until 2026-07-02 16:56:58.94437806 +0000 UTC m=+6.133695096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs") pod "network-metrics-daemon-tr57l" (UID: "0535e4ec-90d9-47eb-9cd6-9f1e9225a103") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:56:56.944929 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:56.944321 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 16:56:56.944929 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:56.944797 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret podName:8ffdd0d1-7f10-427b-8ed5-5b877f73840e nodeName:}" failed. No retries permitted until 2026-07-02 16:56:58.944776776 +0000 UTC m=+6.134093786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret") pod "global-pull-secret-syncer-rn2wj" (UID: "8ffdd0d1-7f10-427b-8ed5-5b877f73840e") : object "kube-system"/"original-pull-secret" not registered Jul 02 16:56:57.045551 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.045465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:56:57.045768 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:57.045712 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 16:56:57.045768 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:57.045738 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 16:56:57.045768 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:57.045753 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9bps9 for pod openshift-network-diagnostics/network-check-target-g6mlh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:56:57.045988 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:57.045859 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9 podName:5055abe2-4719-4230-91ac-e32ddeab3e0d nodeName:}" failed. No retries permitted until 2026-07-02 16:56:59.045839589 +0000 UTC m=+6.235156604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bps9" (UniqueName: "kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9") pod "network-check-target-g6mlh" (UID: "5055abe2-4719-4230-91ac-e32ddeab3e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:56:57.221167 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.220974 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 16:51:55 +0000 UTC" deadline="2027-12-16 12:11:34.042514328 +0000 UTC" Jul 02 16:56:57.221167 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.221042 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12763h14m36.821477445s" Jul 02 16:56:57.264479 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.264439 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:57.264648 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:57.264582 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:56:57.265047 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.265028 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:57.265164 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:57.265142 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:56:57.265234 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.265221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:56:57.265320 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:57.265299 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:56:57.298306 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.298148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerStarted","Data":"3aaa7776f9a6ec3696391f8ed0a1a1991e3be381edf7eabd6cd3b6115c8e037f"} Jul 02 16:56:57.305093 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.305058 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" event={"ID":"2fc75340-7358-46c0-882d-daf8b446a05c","Type":"ContainerStarted","Data":"8dd0a01ae970fbaca2d3981c210c02086a4c0a667290df8aa1473c23b90e2697"} Jul 02 16:56:57.323193 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.323124 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xn7sv" event={"ID":"78775583-a909-469d-ac1c-8ff36fa51ccc","Type":"ContainerStarted","Data":"b9ab62ee7502a8b583d79ce9029ee9b432fd956b8b76cb054748ac75a923bf39"} Jul 02 16:56:57.331482 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.331447 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" event={"ID":"282f21947ce88ed32d4f969496593dd1","Type":"ContainerStarted","Data":"363c0cdf3b75ceebaaf5c2e65ce6ac8f7ad38a75f5515d65248934fe9db1a650"} Jul 02 16:56:57.341899 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.341819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bwj5n" event={"ID":"ab2a2297-66d2-4a77-8e4c-b07cbe46725c","Type":"ContainerStarted","Data":"419a8e2d5eff7896393b1ebc1eef82574b748e6442dcf992f21aba28eb24a236"} Jul 02 16:56:57.347995 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.347534 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-244.ec2.internal" podStartSLOduration=2.3475129949999998 podStartE2EDuration="2.347512995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:56:57.346597606 +0000 UTC m=+4.535914641" watchObservedRunningTime="2026-07-02 16:56:57.347512995 +0000 UTC m=+4.536830031" Jul 02 16:56:57.348764 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.348702 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-925x8" event={"ID":"57708e87-f539-4125-b007-dd9026657fc5","Type":"ContainerStarted","Data":"3b4e42100d91cd7c053da12038308250cb23528c293e5cc770cb58c2f27774e7"} Jul 02 16:56:57.353933 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.353829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"ac78162f4ce7b292091c6641273bd67df8bcfcafa8b7117d7de5bbd1901ef92a"} Jul 02 16:56:57.367492 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.367410 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pfmsz" event={"ID":"dc678e74-bdb7-47dd-8302-5cfc86a9864b","Type":"ContainerStarted","Data":"2a30c514019c29f6854dbced817a816a04d84634e5aa9dc601973f1c04289b5e"} Jul 02 16:56:57.372586 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.372535 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cnkgs" event={"ID":"ed13545a-affc-489d-a843-24e0e2344de6","Type":"ContainerStarted","Data":"a8b0ee07231f13dba082624d186490a224195e0e196f0e9a398ca44d4a96d54a"} Jul 02 16:56:57.387121 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.387085 2582 generic.go:356] "Generic (PLEG): container finished" podID="7324e3503b6cdcf77c0c7de22f620921" containerID="4dc19ab59b0ced27ecbe3546fd035d97e1b71f724cbfec13cc1df270cbb56060" exitCode=0 Jul 02 16:56:57.387411 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.387386 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" event={"ID":"7324e3503b6cdcf77c0c7de22f620921","Type":"ContainerDied","Data":"4dc19ab59b0ced27ecbe3546fd035d97e1b71f724cbfec13cc1df270cbb56060"} Jul 02 16:56:57.402697 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:57.402623 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bx746" event={"ID":"36ccebec-913b-4330-8a79-43988fae2af6","Type":"ContainerStarted","Data":"74ed19a2506386770401dcf828f1d08d7a21007ef323d233bebe6a62103df697"} Jul 02 16:56:58.427263 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:58.427215 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" event={"ID":"7324e3503b6cdcf77c0c7de22f620921","Type":"ContainerStarted","Data":"4d86de4c681f42e6c91dbc6ae3492dd23ae1d33807ccd51c8c1ce25a3f9a2c88"} Jul 02 16:56:58.444136 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:58.442494 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-244.ec2.internal" podStartSLOduration=3.442470549 podStartE2EDuration="3.442470549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:56:58.441969777 +0000 UTC m=+5.631286811" watchObservedRunningTime="2026-07-02 16:56:58.442470549 +0000 UTC m=+5.631787587" Jul 02 16:56:58.964050 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:58.963105 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:58.964050 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:58.963183 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:58.964050 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:58.963356 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 16:56:58.964050 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:58.963423 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret podName:8ffdd0d1-7f10-427b-8ed5-5b877f73840e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:02.963403946 +0000 UTC m=+10.152720978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret") pod "global-pull-secret-syncer-rn2wj" (UID: "8ffdd0d1-7f10-427b-8ed5-5b877f73840e") : object "kube-system"/"original-pull-secret" not registered Jul 02 16:56:58.964050 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:58.963902 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:56:58.964050 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:58.963962 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs podName:0535e4ec-90d9-47eb-9cd6-9f1e9225a103 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:02.963942048 +0000 UTC m=+10.153259077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs") pod "network-metrics-daemon-tr57l" (UID: "0535e4ec-90d9-47eb-9cd6-9f1e9225a103") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:56:59.064050 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:59.064009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:56:59.064262 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:59.064215 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 16:56:59.064262 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:59.064236 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 16:56:59.064262 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:59.064249 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9bps9 for pod openshift-network-diagnostics/network-check-target-g6mlh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:56:59.064426 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:59.064321 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9 podName:5055abe2-4719-4230-91ac-e32ddeab3e0d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:03.064301012 +0000 UTC m=+10.253618044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bps9" (UniqueName: "kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9") pod "network-check-target-g6mlh" (UID: "5055abe2-4719-4230-91ac-e32ddeab3e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:56:59.263068 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:59.262981 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:56:59.263229 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:59.263126 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:56:59.263555 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:59.263534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:56:59.263663 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:59.263638 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:56:59.263733 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:56:59.263718 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:56:59.263844 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:56:59.263826 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:01.261081 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:01.260503 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:01.261081 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:01.260529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:01.261081 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:01.260654 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:01.261081 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:01.260680 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:01.261081 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:01.260830 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:01.261081 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:01.260924 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:02.995941 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:02.995894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:02.996599 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:02.995979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:02.996599 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:02.996132 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 16:57:02.996599 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:02.996204 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret podName:8ffdd0d1-7f10-427b-8ed5-5b877f73840e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:10.996181476 +0000 UTC m=+18.185498511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret") pod "global-pull-secret-syncer-rn2wj" (UID: "8ffdd0d1-7f10-427b-8ed5-5b877f73840e") : object "kube-system"/"original-pull-secret" not registered Jul 02 16:57:02.996854 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:02.996646 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:57:02.996854 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:02.996703 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs podName:0535e4ec-90d9-47eb-9cd6-9f1e9225a103 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:10.996683997 +0000 UTC m=+18.186001026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs") pod "network-metrics-daemon-tr57l" (UID: "0535e4ec-90d9-47eb-9cd6-9f1e9225a103") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:57:03.097555 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:03.097420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:03.097754 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:03.097621 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 16:57:03.097754 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:03.097653 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 16:57:03.097754 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:03.097678 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9bps9 for pod openshift-network-diagnostics/network-check-target-g6mlh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:57:03.097957 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:03.097766 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9 podName:5055abe2-4719-4230-91ac-e32ddeab3e0d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:11.09774497 +0000 UTC m=+18.287062004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bps9" (UniqueName: "kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9") pod "network-check-target-g6mlh" (UID: "5055abe2-4719-4230-91ac-e32ddeab3e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:57:03.261108 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:03.260674 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:03.261108 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:03.260832 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:03.261108 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:03.260841 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:03.261108 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:03.260912 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:03.261108 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:03.260959 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:03.261108 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:03.261024 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:05.259757 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:05.259641 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:05.259757 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:05.259676 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:05.259757 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:05.259644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:05.260356 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:05.259793 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:05.260356 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:05.259947 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:05.260356 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:05.260010 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:07.260554 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:07.260510 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:07.260554 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:07.260527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:07.260996 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:07.260641 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:07.260996 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:07.260638 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:07.260996 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:07.260781 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:07.260996 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:07.260892 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:09.260274 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:09.260214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:09.260274 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:09.260230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:09.260274 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:09.260216 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:09.260816 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:09.260343 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:09.260816 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:09.260398 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:09.260816 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:09.260473 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:11.059944 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:11.059914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:11.060430 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:11.059959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:11.060430 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.060077 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 16:57:11.060430 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.060126 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret podName:8ffdd0d1-7f10-427b-8ed5-5b877f73840e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:27.060112881 +0000 UTC m=+34.249429890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret") pod "global-pull-secret-syncer-rn2wj" (UID: "8ffdd0d1-7f10-427b-8ed5-5b877f73840e") : object "kube-system"/"original-pull-secret" not registered Jul 02 16:57:11.060430 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.060077 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:57:11.060430 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.060236 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs podName:0535e4ec-90d9-47eb-9cd6-9f1e9225a103 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:27.060215648 +0000 UTC m=+34.249532674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs") pod "network-metrics-daemon-tr57l" (UID: "0535e4ec-90d9-47eb-9cd6-9f1e9225a103") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:57:11.160984 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:11.160946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:11.161133 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.161115 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 16:57:11.161190 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.161137 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 16:57:11.161190 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.161148 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9bps9 for pod openshift-network-diagnostics/network-check-target-g6mlh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:57:11.161251 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.161201 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9 podName:5055abe2-4719-4230-91ac-e32ddeab3e0d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:27.161183381 +0000 UTC m=+34.350500389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bps9" (UniqueName: "kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9") pod "network-check-target-g6mlh" (UID: "5055abe2-4719-4230-91ac-e32ddeab3e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:57:11.260423 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:11.260392 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:11.260596 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:11.260394 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:11.260596 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.260513 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:11.260596 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:11.260529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:11.260768 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.260618 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:11.260768 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:11.260705 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:13.261033 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:13.261004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:13.261506 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:13.261118 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:13.261506 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:13.261165 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:13.261506 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:13.261255 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:13.261506 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:13.261292 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:13.261506 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:13.261353 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:15.260615 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:15.260581 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:15.261120 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:15.260640 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:15.261120 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:15.260775 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:15.261233 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:15.261163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:15.261286 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:15.261251 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:15.261371 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:15.261339 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:16.469677 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.469433 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bx746" event={"ID":"36ccebec-913b-4330-8a79-43988fae2af6","Type":"ContainerStarted","Data":"fe15f964edbf664723776ff6a4a2368f579236626cb265737780366b23ff933d"} Jul 02 16:57:16.471272 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.471243 2582 generic.go:356] "Generic (PLEG): container finished" podID="e1e076a9-2f26-4006-ad54-cd2d108d34de" containerID="4b06b94eb2cf3ca9f33757bc572c2185383c379f111bcbe06a45a812559ed2bf" exitCode=0 Jul 02 16:57:16.471400 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.471320 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerDied","Data":"4b06b94eb2cf3ca9f33757bc572c2185383c379f111bcbe06a45a812559ed2bf"} Jul 02 16:57:16.473790 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.473762 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" event={"ID":"2fc75340-7358-46c0-882d-daf8b446a05c","Type":"ContainerStarted","Data":"44bbe09e27106d8d7bee41f594fd1c84f41f39d6026bfc1a4cb274941d2a6d01"} Jul 02 16:57:16.475584 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.475561 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bwj5n" event={"ID":"ab2a2297-66d2-4a77-8e4c-b07cbe46725c","Type":"ContainerStarted","Data":"6931441f0acea70cdcdd81ead24d829532e94bf0a434a138e1089d0684d533d5"} Jul 02 16:57:16.477633 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.477607 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-925x8" event={"ID":"57708e87-f539-4125-b007-dd9026657fc5","Type":"ContainerStarted","Data":"9c6f7382c0b44276fc507d3f9da139ef0bdc3039c2247b2efb3d66bb95cbd93d"} Jul 02 16:57:16.479867 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.479851 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 16:57:16.480215 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.480194 2582 generic.go:356] "Generic (PLEG): container finished" podID="51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd" containerID="83af5c62ba8dd5a321b0d0375a51980a3e6b8c8efa025029de4f475b272549af" exitCode=1 Jul 02 16:57:16.480294 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.480273 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"94f40ebe7ead248d5708f86efc13bea4eec4410cf4dadc9e54bcb48e2150891d"} Jul 02 16:57:16.480364 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.480328 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerDied","Data":"83af5c62ba8dd5a321b0d0375a51980a3e6b8c8efa025029de4f475b272549af"} Jul 02 16:57:16.480459 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.480389 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"82c2f35b3f9d0b0cfc2ae1c27e10626b2d3b0c9b06013dc8af3a46b71ed97db1"} Jul 02 16:57:16.481695 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.481674 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pfmsz" event={"ID":"dc678e74-bdb7-47dd-8302-5cfc86a9864b","Type":"ContainerStarted","Data":"8e7ee2f98d4a936161eb554504ee707441d9db60e02b7b3e813515102c0352e4"} Jul 02 16:57:16.483387 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.483364 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cnkgs" event={"ID":"ed13545a-affc-489d-a843-24e0e2344de6","Type":"ContainerStarted","Data":"951ddc4491b6ebf9066c63c95e5b4ca976e327cc8708fa87da2cf811e15715da"} Jul 02 16:57:16.484472 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.484425 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bx746" podStartSLOduration=10.958785906 podStartE2EDuration="22.48440824s" totalImagesPullingTime="11.525622334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:16.483921508 +0000 UTC m=+23.673238540" watchObservedRunningTime="2026-07-02 16:57:16.48440824 +0000 UTC m=+23.673725276" Jul 02 16:57:16.496890 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.496759 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bwj5n" podStartSLOduration=2.944380791 podStartE2EDuration="22.496745896s" totalImagesPullingTime="19.552365105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:16.49658063 +0000 UTC m=+23.685897663" watchObservedRunningTime="2026-07-02 16:57:16.496745896 +0000 UTC m=+23.686062928" Jul 02 16:57:16.513868 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.513829 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cnkgs" podStartSLOduration=2.892512067 podStartE2EDuration="22.513815639s" totalImagesPullingTime="19.621303572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:16.513456659 +0000 UTC m=+23.702773684" watchObservedRunningTime="2026-07-02 16:57:16.513815639 +0000 UTC m=+23.703132669" Jul 02 16:57:16.531769 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.531700 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-925x8" podStartSLOduration=2.970913458 podStartE2EDuration="22.531679626s" totalImagesPullingTime="19.560766168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:16.531058166 +0000 UTC m=+23.720375198" watchObservedRunningTime="2026-07-02 16:57:16.531679626 +0000 UTC m=+23.720996658" Jul 02 16:57:16.545250 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:16.545208 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pfmsz" podStartSLOduration=2.982293185 podStartE2EDuration="22.54519591s" totalImagesPullingTime="19.562902725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:16.544685534 +0000 UTC m=+23.734002565" watchObservedRunningTime="2026-07-02 16:57:16.54519591 +0000 UTC m=+23.734512940" Jul 02 16:57:17.259928 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:17.259842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:17.259928 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:17.259890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:17.260257 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:17.259986 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:17.260257 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:17.260062 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:17.260257 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:17.260082 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:17.260257 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:17.260172 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:17.492122 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:17.492033 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 16:57:17.493233 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:17.492606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"51d703fb2f8ce620460c356340dcda4c9f5291d1d19215e2b25c19ee04c3212e"} Jul 02 16:57:17.493233 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:17.492764 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"86195ec2437de66dfc805ae09adc3c1fbe76b06612f0ca264475dae5a140e3f3"} Jul 02 16:57:17.493233 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:17.492824 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"734fc322eb8c38322af89d4cbc6b5171f0e7349bc5d5f95df39ce67e126d3592"} Jul 02 16:57:17.542106 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:17.542078 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 16:57:18.261807 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:18.261674 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T16:57:17.542099587Z","UUID":"23ae0fe8-1d54-4b91-8721-b3636c7b00c8","Handler":null,"Name":"","Endpoint":""} Jul 02 16:57:18.263771 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:18.263742 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 16:57:18.263771 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:18.263777 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 16:57:18.497692 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:18.497653 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" event={"ID":"2fc75340-7358-46c0-882d-daf8b446a05c","Type":"ContainerStarted","Data":"41002e05d095162f3aecfdb47918a1618368c3ead1730d24c787b5ea38b3cf93"} Jul 02 16:57:18.499374 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:18.499350 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xn7sv" event={"ID":"78775583-a909-469d-ac1c-8ff36fa51ccc","Type":"ContainerStarted","Data":"3a4d76389bdda4bed3bca1c0918ad63a047628286694eb81e6211d6508abaee1"} Jul 02 16:57:19.260487 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:19.260381 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:19.260487 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:19.260406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:19.260702 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:19.260549 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:19.260786 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:19.260697 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:19.260978 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:19.260858 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:19.260978 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:19.260941 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:19.423482 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:19.423447 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bx746" Jul 02 16:57:19.424096 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:19.424067 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bx746" Jul 02 16:57:19.444859 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:19.444810 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xn7sv" podStartSLOduration=5.875625786 podStartE2EDuration="25.444794692s" totalImagesPullingTime="19.569168906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:18.518166498 +0000 UTC m=+25.707483530" watchObservedRunningTime="2026-07-02 16:57:19.444794692 +0000 UTC m=+26.634111769" Jul 02 16:57:20.507646 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:20.507509 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 16:57:20.507992 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:20.507972 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"4f81f0efe6e5e01aaec42c803471b8eecbdd3a48bff91297c449a91021383329"} Jul 02 16:57:21.259839 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:21.259801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:21.259839 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:21.259841 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:21.260060 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:21.259810 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:21.260060 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:21.259906 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:21.260060 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:21.259997 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:21.260060 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:21.260041 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:21.511588 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:21.511507 2582 generic.go:356] "Generic (PLEG): container finished" podID="e1e076a9-2f26-4006-ad54-cd2d108d34de" containerID="fb85f45c7c51d891078231f5f01535a19762c140701db53a0725894404b1352d" exitCode=0 Jul 02 16:57:21.512220 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:21.511588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerDied","Data":"fb85f45c7c51d891078231f5f01535a19762c140701db53a0725894404b1352d"} Jul 02 16:57:21.513967 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:21.513945 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" event={"ID":"2fc75340-7358-46c0-882d-daf8b446a05c","Type":"ContainerStarted","Data":"71123c5d0fcf7adc97e4f90c07dd498e5b78210936700a424a96251d10d9a987"} Jul 02 16:57:21.546508 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:21.546469 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxq6w" podStartSLOduration=5.598382885 podStartE2EDuration="27.546456105s" totalImagesPullingTime="21.94807322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:21.546249373 +0000 UTC m=+28.735566404" watchObservedRunningTime="2026-07-02 16:57:21.546456105 +0000 UTC m=+28.735773135" Jul 02 16:57:23.260556 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.260280 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:23.260556 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.260360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:23.261177 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:23.260548 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:23.261177 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.260377 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:23.261177 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:23.260622 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:23.261177 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:23.260718 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:23.521462 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.521394 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 16:57:23.521789 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.521759 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"bb4db109c29da1a6e1cb5fc657663010222af64085e4dfec0fbfd836a69f80fb"} Jul 02 16:57:23.522020 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.521998 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:57:23.522226 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.522210 2582 scope.go:117] "RemoveContainer" containerID="83af5c62ba8dd5a321b0d0375a51980a3e6b8c8efa025029de4f475b272549af" Jul 02 16:57:23.524045 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.523970 2582 generic.go:356] "Generic (PLEG): container finished" podID="e1e076a9-2f26-4006-ad54-cd2d108d34de" containerID="043ad89f28814b5c8f82faaefb8bf1a09fcf0394f0fdd2c38229f010d284f932" exitCode=0 Jul 02 16:57:23.524045 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.523998 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerDied","Data":"043ad89f28814b5c8f82faaefb8bf1a09fcf0394f0fdd2c38229f010d284f932"} Jul 02 16:57:23.537996 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:23.537976 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:57:24.529755 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.529698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerStarted","Data":"ccbb5d026410eb5b4927f23fe64ff405338f225e872554246fde7c9e75829bec"} Jul 02 16:57:24.537916 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.537897 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 16:57:24.538294 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.538272 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" event={"ID":"51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd","Type":"ContainerStarted","Data":"0f4862ac2d17e83675c92715fac40a7c65ef4e238817ee4cd52a5ed4d6ecfaf7"} Jul 02 16:57:24.538744 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.538701 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:57:24.538969 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.538948 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:57:24.553949 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.553927 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:57:24.555043 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.555025 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bx746" Jul 02 16:57:24.555133 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.555121 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 02 16:57:24.555613 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.555593 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bx746" Jul 02 16:57:24.577818 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.577621 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" podStartSLOduration=10.959279979 podStartE2EDuration="30.57760443s" totalImagesPullingTime="19.618324451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:24.576594814 +0000 UTC m=+31.765911844" watchObservedRunningTime="2026-07-02 16:57:24.57760443 +0000 UTC m=+31.766921461" Jul 02 16:57:24.741172 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.741124 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tr57l"] Jul 02 16:57:24.741336 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.741285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:24.741430 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:24.741406 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:24.741881 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.741858 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6mlh"] Jul 02 16:57:24.741977 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.741962 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:24.742076 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:24.742056 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:24.742950 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.742929 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rn2wj"] Jul 02 16:57:24.743050 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:24.743036 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:24.743164 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:24.743140 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:25.543092 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:25.543057 2582 generic.go:356] "Generic (PLEG): container finished" podID="e1e076a9-2f26-4006-ad54-cd2d108d34de" containerID="ccbb5d026410eb5b4927f23fe64ff405338f225e872554246fde7c9e75829bec" exitCode=0 Jul 02 16:57:25.543547 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:25.543143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerDied","Data":"ccbb5d026410eb5b4927f23fe64ff405338f225e872554246fde7c9e75829bec"} Jul 02 16:57:26.260216 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:26.260178 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:26.260409 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:26.260224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:26.260409 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:26.260224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:26.260409 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:26.260306 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:26.260567 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:26.260418 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:26.260567 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:26.260516 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:27.081893 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:27.081858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:27.082338 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:27.081904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:27.082338 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:27.082005 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 16:57:27.082338 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:27.082008 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:57:27.082338 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:27.082057 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret podName:8ffdd0d1-7f10-427b-8ed5-5b877f73840e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:59.082043812 +0000 UTC m=+66.271360821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret") pod "global-pull-secret-syncer-rn2wj" (UID: "8ffdd0d1-7f10-427b-8ed5-5b877f73840e") : object "kube-system"/"original-pull-secret" not registered Jul 02 16:57:27.082338 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:27.082070 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs podName:0535e4ec-90d9-47eb-9cd6-9f1e9225a103 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:59.082063909 +0000 UTC m=+66.271380922 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs") pod "network-metrics-daemon-tr57l" (UID: "0535e4ec-90d9-47eb-9cd6-9f1e9225a103") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 16:57:27.182752 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:27.182708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:27.182919 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:27.182888 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 16:57:27.182919 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:27.182915 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 16:57:27.183038 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:27.182925 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9bps9 for pod openshift-network-diagnostics/network-check-target-g6mlh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:57:27.183038 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:27.182980 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9 podName:5055abe2-4719-4230-91ac-e32ddeab3e0d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:59.182965814 +0000 UTC m=+66.372282822 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bps9" (UniqueName: "kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9") pod "network-check-target-g6mlh" (UID: "5055abe2-4719-4230-91ac-e32ddeab3e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 16:57:28.260435 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:28.260402 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:28.260435 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:28.260430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:28.261038 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:28.260491 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:28.261038 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:28.260505 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6mlh" podUID="5055abe2-4719-4230-91ac-e32ddeab3e0d" Jul 02 16:57:28.261038 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:28.260593 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr57l" podUID="0535e4ec-90d9-47eb-9cd6-9f1e9225a103" Jul 02 16:57:28.261038 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:28.260669 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn2wj" podUID="8ffdd0d1-7f10-427b-8ed5-5b877f73840e" Jul 02 16:57:29.127351 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.127277 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-244.ec2.internal" event="NodeReady" Jul 02 16:57:29.127497 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.127418 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 16:57:29.163274 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.163235 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f77ff489c-7d5tp"] Jul 02 16:57:29.165916 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.165890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.168117 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.168087 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 16:57:29.168393 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.168377 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7hkvr\"" Jul 02 16:57:29.168484 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.168457 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 16:57:29.168484 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.168471 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 16:57:29.174102 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.173929 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 16:57:29.177475 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.176190 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n99hl"] Jul 02 16:57:29.178777 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.178579 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6vwwl"] Jul 02 16:57:29.178921 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.178885 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.181662 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.181641 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f77ff489c-7d5tp"] Jul 02 16:57:29.181802 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.181759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6vwwl" Jul 02 16:57:29.182204 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.182176 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 16:57:29.182320 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.182300 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 16:57:29.182320 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.182311 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8ljz5\"" Jul 02 16:57:29.183813 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.183792 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 16:57:29.183813 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.183807 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-624v7\"" Jul 02 16:57:29.183980 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.183931 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 16:57:29.184128 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.184102 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 16:57:29.186615 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.186597 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6vwwl"] Jul 02 16:57:29.190093 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.190071 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n99hl"] Jul 02 16:57:29.199544 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.199518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29934c88-bff5-40b7-a9d7-d63339318a6d-ca-trust-extracted\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.199663 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.199594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-certificates\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.199663 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.199645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-bound-sa-token\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.199663 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.199663 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.199893 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.199716 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-trusted-ca\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.199893 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.199768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-image-registry-private-configuration\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.199893 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.199796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-installation-pull-secrets\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.199893 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.199822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77zg\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-kube-api-access-p77zg\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.300617 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300580 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-bound-sa-token\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.300617 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300661 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0edd7266-95a1-45ad-bc52-caca04bd572f-tmp-dir\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300718 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccm4\" (UniqueName: \"kubernetes.io/projected/ec2ef2cf-1d58-4d0f-83bb-3434f314b777-kube-api-access-7ccm4\") pod \"ingress-canary-6vwwl\" (UID: \"ec2ef2cf-1d58-4d0f-83bb-3434f314b777\") " pod="openshift-ingress-canary/ingress-canary-6vwwl" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec2ef2cf-1d58-4d0f-83bb-3434f314b777-cert\") pod \"ingress-canary-6vwwl\" (UID: \"ec2ef2cf-1d58-4d0f-83bb-3434f314b777\") " pod="openshift-ingress-canary/ingress-canary-6vwwl" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.300774 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.300793 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f77ff489c-7d5tp: secret "image-registry-tls" not found Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-trusted-ca\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300837 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bbc\" (UniqueName: \"kubernetes.io/projected/0edd7266-95a1-45ad-bc52-caca04bd572f-kube-api-access-n4bbc\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.300870 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls podName:29934c88-bff5-40b7-a9d7-d63339318a6d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:29.800848643 +0000 UTC m=+36.990165670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls") pod "image-registry-6f77ff489c-7d5tp" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d") : secret "image-registry-tls" not found Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-image-registry-private-configuration\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.300981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-installation-pull-secrets\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.301007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77zg\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-kube-api-access-p77zg\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.301041 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29934c88-bff5-40b7-a9d7-d63339318a6d-ca-trust-extracted\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.301057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-certificates\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.301080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0edd7266-95a1-45ad-bc52-caca04bd572f-config-volume\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.301905 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.301587 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-certificates\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301905 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.301797 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-trusted-ca\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.301994 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.301912 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29934c88-bff5-40b7-a9d7-d63339318a6d-ca-trust-extracted\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.305552 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.305528 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-image-registry-private-configuration\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.305674 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.305537 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-installation-pull-secrets\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.309034 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.309015 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-bound-sa-token\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.309339 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.309319 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77zg\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-kube-api-access-p77zg\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.401586 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.401502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0edd7266-95a1-45ad-bc52-caca04bd572f-tmp-dir\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.401586 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.401554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccm4\" (UniqueName: \"kubernetes.io/projected/ec2ef2cf-1d58-4d0f-83bb-3434f314b777-kube-api-access-7ccm4\") pod \"ingress-canary-6vwwl\" (UID: \"ec2ef2cf-1d58-4d0f-83bb-3434f314b777\") " pod="openshift-ingress-canary/ingress-canary-6vwwl" Jul 02 16:57:29.401586 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.401578 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec2ef2cf-1d58-4d0f-83bb-3434f314b777-cert\") pod \"ingress-canary-6vwwl\" (UID: \"ec2ef2cf-1d58-4d0f-83bb-3434f314b777\") " pod="openshift-ingress-canary/ingress-canary-6vwwl" Jul 02 16:57:29.401880 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.401608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bbc\" (UniqueName: \"kubernetes.io/projected/0edd7266-95a1-45ad-bc52-caca04bd572f-kube-api-access-n4bbc\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.401880 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.401642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0edd7266-95a1-45ad-bc52-caca04bd572f-config-volume\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.401880 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.401785 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.402026 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.401923 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 16:57:29.402026 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.401982 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls podName:0edd7266-95a1-45ad-bc52-caca04bd572f nodeName:}" failed. No retries permitted until 2026-07-02 16:57:29.901963648 +0000 UTC m=+37.091280659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls") pod "dns-default-n99hl" (UID: "0edd7266-95a1-45ad-bc52-caca04bd572f") : secret "dns-default-metrics-tls" not found Jul 02 16:57:29.402026 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.401999 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0edd7266-95a1-45ad-bc52-caca04bd572f-tmp-dir\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.402259 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.402243 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0edd7266-95a1-45ad-bc52-caca04bd572f-config-volume\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.404519 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.404497 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec2ef2cf-1d58-4d0f-83bb-3434f314b777-cert\") pod \"ingress-canary-6vwwl\" (UID: \"ec2ef2cf-1d58-4d0f-83bb-3434f314b777\") " pod="openshift-ingress-canary/ingress-canary-6vwwl" Jul 02 16:57:29.410369 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.410343 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bbc\" (UniqueName: \"kubernetes.io/projected/0edd7266-95a1-45ad-bc52-caca04bd572f-kube-api-access-n4bbc\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.411043 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.411025 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccm4\" (UniqueName: \"kubernetes.io/projected/ec2ef2cf-1d58-4d0f-83bb-3434f314b777-kube-api-access-7ccm4\") pod \"ingress-canary-6vwwl\" (UID: \"ec2ef2cf-1d58-4d0f-83bb-3434f314b777\") " pod="openshift-ingress-canary/ingress-canary-6vwwl" Jul 02 16:57:29.501042 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.501008 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6vwwl" Jul 02 16:57:29.678439 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.678209 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6vwwl"] Jul 02 16:57:29.682109 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:57:29.682079 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2ef2cf_1d58_4d0f_83bb_3434f314b777.slice/crio-8a5618a9a8e9b94bd1b638f5305173bd09ce7ee2eca8aae0d121e8247a9acc2f WatchSource:0}: Error finding container 8a5618a9a8e9b94bd1b638f5305173bd09ce7ee2eca8aae0d121e8247a9acc2f: Status 404 returned error can't find the container with id 8a5618a9a8e9b94bd1b638f5305173bd09ce7ee2eca8aae0d121e8247a9acc2f Jul 02 16:57:29.805557 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.805534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:29.805710 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.805688 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 16:57:29.805762 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.805711 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f77ff489c-7d5tp: secret "image-registry-tls" not found Jul 02 16:57:29.805800 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.805774 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls podName:29934c88-bff5-40b7-a9d7-d63339318a6d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:30.805759447 +0000 UTC m=+37.995076457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls") pod "image-registry-6f77ff489c-7d5tp" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d") : secret "image-registry-tls" not found Jul 02 16:57:29.907021 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:29.906941 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:29.907109 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.907083 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 16:57:29.907160 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:29.907150 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls podName:0edd7266-95a1-45ad-bc52-caca04bd572f nodeName:}" failed. No retries permitted until 2026-07-02 16:57:30.907134731 +0000 UTC m=+38.096451745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls") pod "dns-default-n99hl" (UID: "0edd7266-95a1-45ad-bc52-caca04bd572f") : secret "dns-default-metrics-tls" not found Jul 02 16:57:30.260379 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.260289 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:30.260379 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.260320 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:30.260379 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.260327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:30.263488 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.263468 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 16:57:30.263610 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.263526 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 16:57:30.264606 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.264426 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 16:57:30.264606 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.264440 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 16:57:30.264606 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.264475 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqd6m\"" Jul 02 16:57:30.264606 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.264430 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pjpx2\"" Jul 02 16:57:30.558760 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.558678 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6vwwl" event={"ID":"ec2ef2cf-1d58-4d0f-83bb-3434f314b777","Type":"ContainerStarted","Data":"8a5618a9a8e9b94bd1b638f5305173bd09ce7ee2eca8aae0d121e8247a9acc2f"} Jul 02 16:57:30.561619 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.561597 2582 generic.go:356] "Generic (PLEG): container finished" podID="e1e076a9-2f26-4006-ad54-cd2d108d34de" containerID="62b6c2ca27327cde1936545e6c7161a5195cd5b163b067c38a3a7d043c38efe6" exitCode=0 Jul 02 16:57:30.561759 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.561653 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerDied","Data":"62b6c2ca27327cde1936545e6c7161a5195cd5b163b067c38a3a7d043c38efe6"} Jul 02 16:57:30.814221 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.814134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:30.814366 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:30.814272 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 16:57:30.814366 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:30.814291 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f77ff489c-7d5tp: secret "image-registry-tls" not found Jul 02 16:57:30.814366 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:30.814363 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls podName:29934c88-bff5-40b7-a9d7-d63339318a6d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:32.814340267 +0000 UTC m=+40.003657277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls") pod "image-registry-6f77ff489c-7d5tp" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d") : secret "image-registry-tls" not found Jul 02 16:57:30.914954 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:30.914915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:30.915125 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:30.915057 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 16:57:30.915185 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:30.915142 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls podName:0edd7266-95a1-45ad-bc52-caca04bd572f nodeName:}" failed. No retries permitted until 2026-07-02 16:57:32.91512317 +0000 UTC m=+40.104440184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls") pod "dns-default-n99hl" (UID: "0edd7266-95a1-45ad-bc52-caca04bd572f") : secret "dns-default-metrics-tls" not found Jul 02 16:57:31.568620 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:31.568575 2582 generic.go:356] "Generic (PLEG): container finished" podID="e1e076a9-2f26-4006-ad54-cd2d108d34de" containerID="5bd60afd6790b222a92fd27afee3bfc889cea9aaea43097dd17524d756a71ed2" exitCode=0 Jul 02 16:57:31.569160 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:31.568656 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerDied","Data":"5bd60afd6790b222a92fd27afee3bfc889cea9aaea43097dd17524d756a71ed2"} Jul 02 16:57:31.952772 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:31.952746 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd"] Jul 02 16:57:31.957980 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:31.957961 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:31.961274 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:31.961254 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 02 16:57:31.961368 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:31.961317 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 02 16:57:31.961416 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:31.961376 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4mxtg\"" Jul 02 16:57:31.970169 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:31.970018 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd"] Jul 02 16:57:32.023820 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.023778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:32.023994 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.023901 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e405fd52-1c06-449f-a1ed-c8f0437ec471-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:32.124756 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.124659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e405fd52-1c06-449f-a1ed-c8f0437ec471-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:32.124756 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.124705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:32.124936 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.124817 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 02 16:57:32.124936 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.124871 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert podName:e405fd52-1c06-449f-a1ed-c8f0437ec471 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:32.624857803 +0000 UTC m=+39.814174812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-5w7nd" (UID: "e405fd52-1c06-449f-a1ed-c8f0437ec471") : secret "networking-console-plugin-cert" not found Jul 02 16:57:32.125481 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.125461 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e405fd52-1c06-449f-a1ed-c8f0437ec471-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:32.573834 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.573799 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" event={"ID":"e1e076a9-2f26-4006-ad54-cd2d108d34de","Type":"ContainerStarted","Data":"8db72fd7c549d2f662545129ff3865fceef525a0e5179c83f758c712053a6c7a"} Jul 02 16:57:32.575165 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.575144 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6vwwl" event={"ID":"ec2ef2cf-1d58-4d0f-83bb-3434f314b777","Type":"ContainerStarted","Data":"2fbed63479799f2d4ab4dbaf5fde4048811ffce26d80797f5e9e43bafa450a3b"} Jul 02 16:57:32.615681 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.615630 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2zrq4" podStartSLOduration=8.822132632 podStartE2EDuration="38.615615641s" totalImagesPullingTime="29.793483009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:32.615279295 +0000 UTC m=+39.804596325" watchObservedRunningTime="2026-07-02 16:57:32.615615641 +0000 UTC m=+39.804932672" Jul 02 16:57:32.629256 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.629226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:32.629388 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.629369 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 02 16:57:32.629452 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.629436 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert podName:e405fd52-1c06-449f-a1ed-c8f0437ec471 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:33.629419928 +0000 UTC m=+40.818736951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-5w7nd" (UID: "e405fd52-1c06-449f-a1ed-c8f0437ec471") : secret "networking-console-plugin-cert" not found Jul 02 16:57:32.641151 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.641113 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6vwwl" podStartSLOduration=1.454934669 podStartE2EDuration="3.641099615s" totalImagesPullingTime="2.186164946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:32.640609197 +0000 UTC m=+39.829926228" watchObservedRunningTime="2026-07-02 16:57:32.641099615 +0000 UTC m=+39.830416667" Jul 02 16:57:32.830463 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.830367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:32.830631 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.830477 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 16:57:32.830631 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.830496 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f77ff489c-7d5tp: secret "image-registry-tls" not found Jul 02 16:57:32.830631 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.830554 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls podName:29934c88-bff5-40b7-a9d7-d63339318a6d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:36.830537626 +0000 UTC m=+44.019854638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls") pod "image-registry-6f77ff489c-7d5tp" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d") : secret "image-registry-tls" not found Jul 02 16:57:32.931385 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:32.931347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:32.931534 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.931496 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 16:57:32.931579 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:32.931559 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls podName:0edd7266-95a1-45ad-bc52-caca04bd572f nodeName:}" failed. No retries permitted until 2026-07-02 16:57:36.931543767 +0000 UTC m=+44.120860776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls") pod "dns-default-n99hl" (UID: "0edd7266-95a1-45ad-bc52-caca04bd572f") : secret "dns-default-metrics-tls" not found Jul 02 16:57:33.637697 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:33.637652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:33.638292 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:33.637847 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 02 16:57:33.638292 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:33.637948 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert podName:e405fd52-1c06-449f-a1ed-c8f0437ec471 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:35.637924451 +0000 UTC m=+42.827241479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-5w7nd" (UID: "e405fd52-1c06-449f-a1ed-c8f0437ec471") : secret "networking-console-plugin-cert" not found Jul 02 16:57:35.650832 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:35.650798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:35.651193 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:35.650924 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 02 16:57:35.651193 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:35.650975 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert podName:e405fd52-1c06-449f-a1ed-c8f0437ec471 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:39.650961294 +0000 UTC m=+46.840278302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-5w7nd" (UID: "e405fd52-1c06-449f-a1ed-c8f0437ec471") : secret "networking-console-plugin-cert" not found Jul 02 16:57:36.860908 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:36.860870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:36.861277 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:36.860989 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 16:57:36.861277 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:36.861002 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f77ff489c-7d5tp: secret "image-registry-tls" not found Jul 02 16:57:36.861277 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:36.861050 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls podName:29934c88-bff5-40b7-a9d7-d63339318a6d nodeName:}" failed. No retries permitted until 2026-07-02 16:57:44.861036365 +0000 UTC m=+52.050353374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls") pod "image-registry-6f77ff489c-7d5tp" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d") : secret "image-registry-tls" not found Jul 02 16:57:36.961418 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:36.961383 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:36.961561 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:36.961526 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 16:57:36.961599 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:36.961586 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls podName:0edd7266-95a1-45ad-bc52-caca04bd572f nodeName:}" failed. No retries permitted until 2026-07-02 16:57:44.961570348 +0000 UTC m=+52.150887357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls") pod "dns-default-n99hl" (UID: "0edd7266-95a1-45ad-bc52-caca04bd572f") : secret "dns-default-metrics-tls" not found Jul 02 16:57:39.385614 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.385579 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5"] Jul 02 16:57:39.419054 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.419019 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5"] Jul 02 16:57:39.419194 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.419139 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.421860 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.421837 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 02 16:57:39.422030 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.422014 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 02 16:57:39.422093 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.422048 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 02 16:57:39.422965 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.422949 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 02 16:57:39.423050 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.423031 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t94w6\"" Jul 02 16:57:39.481597 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.481569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48596f3b-7679-4a0e-9d5f-2e00018bb50a-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.481597 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.481599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48596f3b-7679-4a0e-9d5f-2e00018bb50a-signing-key\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.481795 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.481641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpkzm\" (UniqueName: \"kubernetes.io/projected/48596f3b-7679-4a0e-9d5f-2e00018bb50a-kube-api-access-kpkzm\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.582134 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.582092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpkzm\" (UniqueName: \"kubernetes.io/projected/48596f3b-7679-4a0e-9d5f-2e00018bb50a-kube-api-access-kpkzm\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.582289 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.582178 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48596f3b-7679-4a0e-9d5f-2e00018bb50a-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.582289 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.582196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48596f3b-7679-4a0e-9d5f-2e00018bb50a-signing-key\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.582845 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.582822 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48596f3b-7679-4a0e-9d5f-2e00018bb50a-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.586152 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.586130 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48596f3b-7679-4a0e-9d5f-2e00018bb50a-signing-key\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.593149 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.593126 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpkzm\" (UniqueName: \"kubernetes.io/projected/48596f3b-7679-4a0e-9d5f-2e00018bb50a-kube-api-access-kpkzm\") pod \"service-ca-7bb4f4bcc7-6r7f5\" (UID: \"48596f3b-7679-4a0e-9d5f-2e00018bb50a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.683117 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.683047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:39.683227 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:39.683157 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 02 16:57:39.683227 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:39.683214 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert podName:e405fd52-1c06-449f-a1ed-c8f0437ec471 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:47.683201084 +0000 UTC m=+54.872518093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-5w7nd" (UID: "e405fd52-1c06-449f-a1ed-c8f0437ec471") : secret "networking-console-plugin-cert" not found Jul 02 16:57:39.728107 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.728084 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" Jul 02 16:57:39.868035 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:39.868004 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5"] Jul 02 16:57:39.871856 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:57:39.871830 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48596f3b_7679_4a0e_9d5f_2e00018bb50a.slice/crio-9458fb14cfcd67372abaae006a8a63114471e986379b34691e6eafa5d8879b0f WatchSource:0}: Error finding container 9458fb14cfcd67372abaae006a8a63114471e986379b34691e6eafa5d8879b0f: Status 404 returned error can't find the container with id 9458fb14cfcd67372abaae006a8a63114471e986379b34691e6eafa5d8879b0f Jul 02 16:57:40.596207 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:40.596168 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" event={"ID":"48596f3b-7679-4a0e-9d5f-2e00018bb50a","Type":"ContainerStarted","Data":"9458fb14cfcd67372abaae006a8a63114471e986379b34691e6eafa5d8879b0f"} Jul 02 16:57:42.604028 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:42.603992 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" event={"ID":"48596f3b-7679-4a0e-9d5f-2e00018bb50a","Type":"ContainerStarted","Data":"ccbeeaf2dcba30a319c023f48e56dbed6a632e190901513e713a4e1da33cd8a3"} Jul 02 16:57:42.630796 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:42.627692 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6r7f5" podStartSLOduration=1.120653246 podStartE2EDuration="3.627674574s" totalImagesPullingTime="2.507021328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:42.627249842 +0000 UTC m=+49.816566874" watchObservedRunningTime="2026-07-02 16:57:42.627674574 +0000 UTC m=+49.816991606" Jul 02 16:57:44.924919 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:44.924885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:57:44.925377 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:44.925070 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 16:57:44.925377 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:44.925094 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f77ff489c-7d5tp: secret "image-registry-tls" not found Jul 02 16:57:44.925377 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:44.925172 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls podName:29934c88-bff5-40b7-a9d7-d63339318a6d nodeName:}" failed. No retries permitted until 2026-07-02 16:58:00.92515159 +0000 UTC m=+68.114468605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls") pod "image-registry-6f77ff489c-7d5tp" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d") : secret "image-registry-tls" not found Jul 02 16:57:45.025843 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:45.025808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:57:45.026012 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:45.025972 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 16:57:45.026051 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:45.026035 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls podName:0edd7266-95a1-45ad-bc52-caca04bd572f nodeName:}" failed. No retries permitted until 2026-07-02 16:58:01.026019804 +0000 UTC m=+68.215336813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls") pod "dns-default-n99hl" (UID: "0edd7266-95a1-45ad-bc52-caca04bd572f") : secret "dns-default-metrics-tls" not found Jul 02 16:57:47.745455 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:47.745422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:57:47.745972 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:47.745530 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 02 16:57:47.745972 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:57:47.745811 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert podName:e405fd52-1c06-449f-a1ed-c8f0437ec471 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:03.745783908 +0000 UTC m=+70.935100917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-5w7nd" (UID: "e405fd52-1c06-449f-a1ed-c8f0437ec471") : secret "networking-console-plugin-cert" not found Jul 02 16:57:56.555628 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:56.555599 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2d9jh" Jul 02 16:57:59.130918 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.130880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:59.131424 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.130953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:59.134161 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.134142 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 16:57:59.134874 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.134857 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 16:57:59.143326 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.143302 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ffdd0d1-7f10-427b-8ed5-5b877f73840e-original-pull-secret\") pod \"global-pull-secret-syncer-rn2wj\" (UID: \"8ffdd0d1-7f10-427b-8ed5-5b877f73840e\") " pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:59.143737 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.143704 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0535e4ec-90d9-47eb-9cd6-9f1e9225a103-metrics-certs\") pod \"network-metrics-daemon-tr57l\" (UID: \"0535e4ec-90d9-47eb-9cd6-9f1e9225a103\") " pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:59.231499 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.231466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:59.234410 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.234393 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 16:57:59.245139 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.245116 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 16:57:59.255591 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.255565 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bps9\" (UniqueName: \"kubernetes.io/projected/5055abe2-4719-4230-91ac-e32ddeab3e0d-kube-api-access-9bps9\") pod \"network-check-target-g6mlh\" (UID: \"5055abe2-4719-4230-91ac-e32ddeab3e0d\") " pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:59.375031 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.375001 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqd6m\"" Jul 02 16:57:59.378602 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.378588 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn2wj" Jul 02 16:57:59.382721 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.382707 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:57:59.387271 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.387256 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pjpx2\"" Jul 02 16:57:59.395425 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.395408 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr57l" Jul 02 16:57:59.562364 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.562288 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rn2wj"] Jul 02 16:57:59.578140 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:57:59.578115 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffdd0d1_7f10_427b_8ed5_5b877f73840e.slice/crio-33bab57d3fce4e38a6c26f01f642e0fae0e960793c98988e01d23d4de57be188 WatchSource:0}: Error finding container 33bab57d3fce4e38a6c26f01f642e0fae0e960793c98988e01d23d4de57be188: Status 404 returned error can't find the container with id 33bab57d3fce4e38a6c26f01f642e0fae0e960793c98988e01d23d4de57be188 Jul 02 16:57:59.581242 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.581217 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6mlh"] Jul 02 16:57:59.583663 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:57:59.583641 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5055abe2_4719_4230_91ac_e32ddeab3e0d.slice/crio-89bbbca1f8264d7c91f273f4842b8e1380cda281c0746807699d6b88bb97df3b WatchSource:0}: Error finding container 89bbbca1f8264d7c91f273f4842b8e1380cda281c0746807699d6b88bb97df3b: Status 404 returned error can't find the container with id 89bbbca1f8264d7c91f273f4842b8e1380cda281c0746807699d6b88bb97df3b Jul 02 16:57:59.605269 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.605248 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tr57l"] Jul 02 16:57:59.609348 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:57:59.609317 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0535e4ec_90d9_47eb_9cd6_9f1e9225a103.slice/crio-9d68aaf81db95e7a264706fc97478e75d0ad42042b10b3351c598b6a35e91fae WatchSource:0}: Error finding container 9d68aaf81db95e7a264706fc97478e75d0ad42042b10b3351c598b6a35e91fae: Status 404 returned error can't find the container with id 9d68aaf81db95e7a264706fc97478e75d0ad42042b10b3351c598b6a35e91fae Jul 02 16:57:59.646001 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.645932 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6mlh" event={"ID":"5055abe2-4719-4230-91ac-e32ddeab3e0d","Type":"ContainerStarted","Data":"89bbbca1f8264d7c91f273f4842b8e1380cda281c0746807699d6b88bb97df3b"} Jul 02 16:57:59.647080 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.647057 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tr57l" event={"ID":"0535e4ec-90d9-47eb-9cd6-9f1e9225a103","Type":"ContainerStarted","Data":"9d68aaf81db95e7a264706fc97478e75d0ad42042b10b3351c598b6a35e91fae"} Jul 02 16:57:59.648009 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:57:59.647986 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rn2wj" event={"ID":"8ffdd0d1-7f10-427b-8ed5-5b877f73840e","Type":"ContainerStarted","Data":"33bab57d3fce4e38a6c26f01f642e0fae0e960793c98988e01d23d4de57be188"} Jul 02 16:58:00.945682 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:00.945106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:58:00.950636 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:00.950608 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"image-registry-6f77ff489c-7d5tp\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:58:00.981398 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:00.981368 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7hkvr\"" Jul 02 16:58:00.988886 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:00.988863 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:58:01.045810 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.045757 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:58:01.049253 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.049201 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0edd7266-95a1-45ad-bc52-caca04bd572f-metrics-tls\") pod \"dns-default-n99hl\" (UID: \"0edd7266-95a1-45ad-bc52-caca04bd572f\") " pod="openshift-dns/dns-default-n99hl" Jul 02 16:58:01.160260 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.160231 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f77ff489c-7d5tp"] Jul 02 16:58:01.296287 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.296259 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8ljz5\"" Jul 02 16:58:01.303700 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.303647 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n99hl" Jul 02 16:58:01.493155 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.493061 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n99hl"] Jul 02 16:58:01.497336 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:58:01.497307 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edd7266_95a1_45ad_bc52_caca04bd572f.slice/crio-d93e7beb146f211ce32121915e77d55a838cd95a8ead70e9bb8603ba2d1c1f85 WatchSource:0}: Error finding container d93e7beb146f211ce32121915e77d55a838cd95a8ead70e9bb8603ba2d1c1f85: Status 404 returned error can't find the container with id d93e7beb146f211ce32121915e77d55a838cd95a8ead70e9bb8603ba2d1c1f85 Jul 02 16:58:01.660817 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.660779 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tr57l" event={"ID":"0535e4ec-90d9-47eb-9cd6-9f1e9225a103","Type":"ContainerStarted","Data":"b8126381b70d5f1daae2b9f0e0dd38cc8fcb3c8b91c57def8b17f4934f62adbd"} Jul 02 16:58:01.661007 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.660849 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tr57l" event={"ID":"0535e4ec-90d9-47eb-9cd6-9f1e9225a103","Type":"ContainerStarted","Data":"2161255f5f3ea70dc3adc55e5441e0170969d745da59e53bdd5b51a7828437d2"} Jul 02 16:58:01.669208 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.669105 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n99hl" event={"ID":"0edd7266-95a1-45ad-bc52-caca04bd572f","Type":"ContainerStarted","Data":"d93e7beb146f211ce32121915e77d55a838cd95a8ead70e9bb8603ba2d1c1f85"} Jul 02 16:58:01.671854 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.671824 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" event={"ID":"29934c88-bff5-40b7-a9d7-d63339318a6d","Type":"ContainerStarted","Data":"948d5176d5c192ed1fe3fc114439da1ed480150bfac45a36e59475aff7a50700"} Jul 02 16:58:01.671996 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.671876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" event={"ID":"29934c88-bff5-40b7-a9d7-d63339318a6d","Type":"ContainerStarted","Data":"f131e402e892e0ccb3c0350c9ae1682aac4b5f1fb28ac88ad4bd189412d06486"} Jul 02 16:58:01.672059 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.672013 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:58:01.718159 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.718101 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tr57l" podStartSLOduration=66.503897465 podStartE2EDuration="1m7.718084218s" totalImagesPullingTime="1.214186753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:01.689464288 +0000 UTC m=+68.878781321" watchObservedRunningTime="2026-07-02 16:58:01.718084218 +0000 UTC m=+68.907401249" Jul 02 16:58:01.718352 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:01.718244 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" podStartSLOduration=58.718238212 podStartE2EDuration="58.718238212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:01.714931507 +0000 UTC m=+68.904248540" watchObservedRunningTime="2026-07-02 16:58:01.718238212 +0000 UTC m=+68.907555221" Jul 02 16:58:03.254682 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.254644 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-s2ss9"] Jul 02 16:58:03.256740 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.256689 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l"] Jul 02 16:58:03.257002 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.256972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.258761 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.258742 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.260092 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.260068 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-frtkd\"" Jul 02 16:58:03.269309 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.269290 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-s2ss9"] Jul 02 16:58:03.276119 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.275891 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 02 16:58:03.276119 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.276068 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 02 16:58:03.276552 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.276382 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 02 16:58:03.283451 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.283423 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 02 16:58:03.283613 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.283493 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 02 16:58:03.283701 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.283677 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 02 16:58:03.284098 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.284077 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 16:58:03.284194 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.284113 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 16:58:03.284304 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.284238 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 02 16:58:03.284304 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.284280 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 02 16:58:03.285539 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.285517 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l"] Jul 02 16:58:03.286688 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.286659 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 02 16:58:03.287775 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.287754 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 02 16:58:03.365992 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.365959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.366277 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b402035e-12c0-41f9-94e8-79ae8d9e4d63-tmp\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.366422 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9h4\" (UniqueName: \"kubernetes.io/projected/f9784260-7844-44f5-84d5-fe66721fe078-kube-api-access-vk9h4\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.366560 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-ca\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.366679 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b402035e-12c0-41f9-94e8-79ae8d9e4d63-snapshots\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.366679 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b402035e-12c0-41f9-94e8-79ae8d9e4d63-serving-cert\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.366679 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.366866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366693 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9784260-7844-44f5-84d5-fe66721fe078-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.366866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b402035e-12c0-41f9-94e8-79ae8d9e4d63-service-ca-bundle\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.366866 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-hub\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.367018 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b402035e-12c0-41f9-94e8-79ae8d9e4d63-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.367018 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.366941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4cm5\" (UniqueName: \"kubernetes.io/projected/b402035e-12c0-41f9-94e8-79ae8d9e4d63-kube-api-access-z4cm5\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.379795 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.379766 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f77ff489c-7d5tp"] Jul 02 16:58:03.467368 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9784260-7844-44f5-84d5-fe66721fe078-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.467368 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b402035e-12c0-41f9-94e8-79ae8d9e4d63-service-ca-bundle\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.467608 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-hub\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.467608 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b402035e-12c0-41f9-94e8-79ae8d9e4d63-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.467608 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4cm5\" (UniqueName: \"kubernetes.io/projected/b402035e-12c0-41f9-94e8-79ae8d9e4d63-kube-api-access-z4cm5\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.467608 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.467608 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b402035e-12c0-41f9-94e8-79ae8d9e4d63-tmp\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.467608 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9h4\" (UniqueName: \"kubernetes.io/projected/f9784260-7844-44f5-84d5-fe66721fe078-kube-api-access-vk9h4\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.467608 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-ca\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.467992 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467617 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b402035e-12c0-41f9-94e8-79ae8d9e4d63-snapshots\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.467992 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467645 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b402035e-12c0-41f9-94e8-79ae8d9e4d63-serving-cert\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.467992 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.467672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.468324 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.468175 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9784260-7844-44f5-84d5-fe66721fe078-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.468324 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.468248 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b402035e-12c0-41f9-94e8-79ae8d9e4d63-service-ca-bundle\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.468490 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.468461 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b402035e-12c0-41f9-94e8-79ae8d9e4d63-tmp\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.468583 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.468542 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b402035e-12c0-41f9-94e8-79ae8d9e4d63-snapshots\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.469369 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.469347 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b402035e-12c0-41f9-94e8-79ae8d9e4d63-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.470571 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.470524 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.470999 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.470948 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-hub\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.471139 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.471120 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.471388 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.471333 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f9784260-7844-44f5-84d5-fe66721fe078-ca\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.471716 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.471697 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b402035e-12c0-41f9-94e8-79ae8d9e4d63-serving-cert\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.488952 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.488924 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9h4\" (UniqueName: \"kubernetes.io/projected/f9784260-7844-44f5-84d5-fe66721fe078-kube-api-access-vk9h4\") pod \"cluster-proxy-proxy-agent-57784b5bcb-rhs2l\" (UID: \"f9784260-7844-44f5-84d5-fe66721fe078\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.504762 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.504675 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4cm5\" (UniqueName: \"kubernetes.io/projected/b402035e-12c0-41f9-94e8-79ae8d9e4d63-kube-api-access-z4cm5\") pod \"insights-operator-5f67b55555-s2ss9\" (UID: \"b402035e-12c0-41f9-94e8-79ae8d9e4d63\") " pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.572002 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.571961 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-s2ss9" Jul 02 16:58:03.588989 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.588958 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" Jul 02 16:58:03.770269 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.770185 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:58:03.773256 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:03.773232 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e405fd52-1c06-449f-a1ed-c8f0437ec471-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-5w7nd\" (UID: \"e405fd52-1c06-449f-a1ed-c8f0437ec471\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:58:04.077649 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:04.077560 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4mxtg\"" Jul 02 16:58:04.087773 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:04.087740 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" Jul 02 16:58:04.835665 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:04.835575 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd"] Jul 02 16:58:04.838118 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:58:04.838087 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode405fd52_1c06_449f_a1ed_c8f0437ec471.slice/crio-051b2f0277385c1138110e03eac36e7069f05b130e118f4adb6503e434a81692 WatchSource:0}: Error finding container 051b2f0277385c1138110e03eac36e7069f05b130e118f4adb6503e434a81692: Status 404 returned error can't find the container with id 051b2f0277385c1138110e03eac36e7069f05b130e118f4adb6503e434a81692 Jul 02 16:58:04.858152 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:04.857898 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-s2ss9"] Jul 02 16:58:04.859534 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:04.859505 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l"] Jul 02 16:58:04.864992 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:58:04.864645 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb402035e_12c0_41f9_94e8_79ae8d9e4d63.slice/crio-2625e111ad2393fc57c0b2b288945876883fc1fe380715afc812f1bfa9510b1d WatchSource:0}: Error finding container 2625e111ad2393fc57c0b2b288945876883fc1fe380715afc812f1bfa9510b1d: Status 404 returned error can't find the container with id 2625e111ad2393fc57c0b2b288945876883fc1fe380715afc812f1bfa9510b1d Jul 02 16:58:05.722104 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.722067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n99hl" event={"ID":"0edd7266-95a1-45ad-bc52-caca04bd572f","Type":"ContainerStarted","Data":"d22216c4f22d73792851406d2c589a6d3bc5c9ccfb958317fa5e91d691ef2cf7"} Jul 02 16:58:05.722273 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.722135 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n99hl" event={"ID":"0edd7266-95a1-45ad-bc52-caca04bd572f","Type":"ContainerStarted","Data":"1f7fbf7a1e05b47243f86961c7adfc8a6b1747099eb324b18999272d50373a02"} Jul 02 16:58:05.722962 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.722882 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n99hl" Jul 02 16:58:05.724924 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.724890 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rn2wj" event={"ID":"8ffdd0d1-7f10-427b-8ed5-5b877f73840e","Type":"ContainerStarted","Data":"192fcb4f3a760d817e2db812749504211b5cc212a6bb5eed8799408f5f52e2c1"} Jul 02 16:58:05.727512 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.727477 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" event={"ID":"e405fd52-1c06-449f-a1ed-c8f0437ec471","Type":"ContainerStarted","Data":"051b2f0277385c1138110e03eac36e7069f05b130e118f4adb6503e434a81692"} Jul 02 16:58:05.728873 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.728849 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" event={"ID":"f9784260-7844-44f5-84d5-fe66721fe078","Type":"ContainerStarted","Data":"c647661989ce8ba326c24b437946c41b1cdc7fff92184130c6fdd5c397c64c58"} Jul 02 16:58:05.732176 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.732064 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6mlh" event={"ID":"5055abe2-4719-4230-91ac-e32ddeab3e0d","Type":"ContainerStarted","Data":"b574586da84f1785378c107ed854e455e35079d43a1742c36f2809b9452aab41"} Jul 02 16:58:05.732176 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.732105 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 16:58:05.737673 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.734338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-s2ss9" event={"ID":"b402035e-12c0-41f9-94e8-79ae8d9e4d63","Type":"ContainerStarted","Data":"2625e111ad2393fc57c0b2b288945876883fc1fe380715afc812f1bfa9510b1d"} Jul 02 16:58:05.754907 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.754793 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n99hl" podStartSLOduration=33.586586719 podStartE2EDuration="36.754775298s" totalImagesPullingTime="3.168188579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:05.753620017 +0000 UTC m=+72.942937050" watchObservedRunningTime="2026-07-02 16:58:05.754775298 +0000 UTC m=+72.944092330" Jul 02 16:58:05.839276 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.838609 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rn2wj" podStartSLOduration=66.793587655 podStartE2EDuration="1m11.838589325s" totalImagesPullingTime="5.04500167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:05.838207199 +0000 UTC m=+73.027524229" watchObservedRunningTime="2026-07-02 16:58:05.838589325 +0000 UTC m=+73.027906356" Jul 02 16:58:05.839276 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:05.838769 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g6mlh" podStartSLOduration=66.79821142 podStartE2EDuration="1m11.838756706s" totalImagesPullingTime="5.040545286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:05.803906625 +0000 UTC m=+72.993223659" watchObservedRunningTime="2026-07-02 16:58:05.838756706 +0000 UTC m=+73.028073738" Jul 02 16:58:09.565026 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.564984 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4wdnp"] Jul 02 16:58:09.566995 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.566979 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.571820 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.571263 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 16:58:09.571820 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.571494 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blnmw\"" Jul 02 16:58:09.571820 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.571659 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 16:58:09.639828 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.639788 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wdnp"] Jul 02 16:58:09.723199 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.723163 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.723199 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.723205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-data-volume\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.723397 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.723227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.723397 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.723306 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz99n\" (UniqueName: \"kubernetes.io/projected/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-api-access-nz99n\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.723397 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.723327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-crio-socket\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.752986 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.752950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" event={"ID":"e405fd52-1c06-449f-a1ed-c8f0437ec471","Type":"ContainerStarted","Data":"e2283f9b83267ae700ad07454c7a241c25c8552349bec046776c10b37bcc1975"} Jul 02 16:58:09.755024 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.754992 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" event={"ID":"f9784260-7844-44f5-84d5-fe66721fe078","Type":"ContainerStarted","Data":"e545e9a349b2b6b23759db47f703bcc1b29600eb17a4a5cf4c206d87816cdd00"} Jul 02 16:58:09.755138 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.755035 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" event={"ID":"f9784260-7844-44f5-84d5-fe66721fe078","Type":"ContainerStarted","Data":"5c575d682ca8bbdd0b9dfeea27e8dba6bc806becca3174b4b9472b9e48c0ee3c"} Jul 02 16:58:09.755138 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.755053 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" event={"ID":"f9784260-7844-44f5-84d5-fe66721fe078","Type":"ContainerStarted","Data":"ca39629241324d49e94ebaa8182fc76425804802e76eb159d8a83c12d4ab66da"} Jul 02 16:58:09.756277 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.756255 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-s2ss9" event={"ID":"b402035e-12c0-41f9-94e8-79ae8d9e4d63","Type":"ContainerStarted","Data":"341cd5bdbf31fa7e0e7270b1cf3528f1e218d4a0a4d2a2d67aa1314f056c73be"} Jul 02 16:58:09.778289 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.778243 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-5w7nd" podStartSLOduration=34.750711834 podStartE2EDuration="38.778229541s" totalImagesPullingTime="4.027517707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:09.776124719 +0000 UTC m=+76.965441750" watchObservedRunningTime="2026-07-02 16:58:09.778229541 +0000 UTC m=+76.967546571" Jul 02 16:58:09.815506 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.815399 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-s2ss9" podStartSLOduration=2.809346088 podStartE2EDuration="6.815381023s" totalImagesPullingTime="4.006034935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:09.813866389 +0000 UTC m=+77.003183421" watchObservedRunningTime="2026-07-02 16:58:09.815381023 +0000 UTC m=+77.004698058" Jul 02 16:58:09.824299 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.824270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.824410 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.824311 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-data-volume\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.824410 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.824332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.824410 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.824366 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz99n\" (UniqueName: \"kubernetes.io/projected/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-api-access-nz99n\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.824410 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.824389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-crio-socket\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.824410 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:09.824398 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 16:58:09.824628 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:09.824461 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 16:58:09.824628 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.824464 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-crio-socket\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.824628 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:09.824473 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls podName:35992b57-3bd9-42c2-8287-c50dd2ddc6a9 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:10.324455345 +0000 UTC m=+77.513772373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wdnp" (UID: "35992b57-3bd9-42c2-8287-c50dd2ddc6a9") : secret "insights-runtime-extractor-tls" not found Jul 02 16:58:09.824628 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:09.824566 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm podName:35992b57-3bd9-42c2-8287-c50dd2ddc6a9 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:10.324540868 +0000 UTC m=+77.513857893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wdnp" (UID: "35992b57-3bd9-42c2-8287-c50dd2ddc6a9") : configmap "kube-rbac-proxy" not found Jul 02 16:58:09.824821 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.824662 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-data-volume\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:09.859144 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:09.859110 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz99n\" (UniqueName: \"kubernetes.io/projected/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-api-access-nz99n\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:10.328616 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:10.328579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:10.328616 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:10.328620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:10.328826 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:10.328699 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 16:58:10.328826 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:10.328749 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 16:58:10.328826 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:10.328789 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm podName:35992b57-3bd9-42c2-8287-c50dd2ddc6a9 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:11.328771517 +0000 UTC m=+78.518088528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wdnp" (UID: "35992b57-3bd9-42c2-8287-c50dd2ddc6a9") : configmap "kube-rbac-proxy" not found Jul 02 16:58:10.328928 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:10.328842 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls podName:35992b57-3bd9-42c2-8287-c50dd2ddc6a9 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:11.328827782 +0000 UTC m=+78.518144795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wdnp" (UID: "35992b57-3bd9-42c2-8287-c50dd2ddc6a9") : secret "insights-runtime-extractor-tls" not found Jul 02 16:58:10.937598 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:10.937570 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n99hl_0edd7266-95a1-45ad-bc52-caca04bd572f/dns/0.log" Jul 02 16:58:11.103073 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:11.103037 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n99hl_0edd7266-95a1-45ad-bc52-caca04bd572f/kube-rbac-proxy/0.log" Jul 02 16:58:11.336278 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:11.336191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:11.336278 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:11.336229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:11.336495 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:11.336302 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 16:58:11.336495 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:11.336364 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 16:58:11.336495 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:11.336366 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm podName:35992b57-3bd9-42c2-8287-c50dd2ddc6a9 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:13.336349927 +0000 UTC m=+80.525666953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wdnp" (UID: "35992b57-3bd9-42c2-8287-c50dd2ddc6a9") : configmap "kube-rbac-proxy" not found Jul 02 16:58:11.336495 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:11.336438 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls podName:35992b57-3bd9-42c2-8287-c50dd2ddc6a9 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:13.336419117 +0000 UTC m=+80.525736133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wdnp" (UID: "35992b57-3bd9-42c2-8287-c50dd2ddc6a9") : secret "insights-runtime-extractor-tls" not found Jul 02 16:58:11.893915 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:11.893890 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pfmsz_dc678e74-bdb7-47dd-8302-5cfc86a9864b/dns-node-resolver/0.log" Jul 02 16:58:12.703821 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:12.703788 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f77ff489c-7d5tp_29934c88-bff5-40b7-a9d7-d63339318a6d/registry/0.log" Jul 02 16:58:12.937617 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:12.937591 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bwj5n_ab2a2297-66d2-4a77-8e4c-b07cbe46725c/node-ca/0.log" Jul 02 16:58:13.352781 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:13.352749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:13.352974 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:13.352799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:13.352974 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:13.352887 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 16:58:13.352974 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:13.352893 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 16:58:13.352974 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:13.352952 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm podName:35992b57-3bd9-42c2-8287-c50dd2ddc6a9 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:17.352935195 +0000 UTC m=+84.542252204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wdnp" (UID: "35992b57-3bd9-42c2-8287-c50dd2ddc6a9") : configmap "kube-rbac-proxy" not found Jul 02 16:58:13.352974 ip-10-0-134-244 kubenswrapper[2582]: E0702 16:58:13.352971 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls podName:35992b57-3bd9-42c2-8287-c50dd2ddc6a9 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:17.35296284 +0000 UTC m=+84.542279850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wdnp" (UID: "35992b57-3bd9-42c2-8287-c50dd2ddc6a9") : secret "insights-runtime-extractor-tls" not found Jul 02 16:58:13.736443 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:13.736409 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6vwwl_ec2ef2cf-1d58-4d0f-83bb-3434f314b777/serve-healthcheck-canary/0.log" Jul 02 16:58:16.744251 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:16.744124 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n99hl" Jul 02 16:58:16.865594 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:16.865530 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57784b5bcb-rhs2l" podStartSLOduration=9.821820516 podStartE2EDuration="13.865510353s" totalImagesPullingTime="4.043689837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:09.890064796 +0000 UTC m=+77.079381829" watchObservedRunningTime="2026-07-02 16:58:16.865510353 +0000 UTC m=+84.054827385" Jul 02 16:58:17.366590 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.366556 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-n4jww"] Jul 02 16:58:17.369239 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.369224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.382013 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.381992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.382124 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.382022 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.382124 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.382050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d30eed73-7ea8-4ead-9662-764a01687571-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.382124 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.382105 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96rn\" (UniqueName: \"kubernetes.io/projected/d30eed73-7ea8-4ead-9662-764a01687571-kube-api-access-h96rn\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.382225 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.382141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d30eed73-7ea8-4ead-9662-764a01687571-metrics-client-ca\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.382225 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.382167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:17.382225 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.382189 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.382225 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.382214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:17.382652 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.382632 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:17.384292 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.384261 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35992b57-3bd9-42c2-8287-c50dd2ddc6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wdnp\" (UID: \"35992b57-3bd9-42c2-8287-c50dd2ddc6a9\") " pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:17.439488 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.439461 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 02 16:58:17.444078 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.444058 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 16:58:17.444792 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.444778 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 02 16:58:17.445291 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.445274 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 16:58:17.445356 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.445291 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 02 16:58:17.445596 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.445578 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-62tt5\"" Jul 02 16:58:17.458214 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.458188 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 16:58:17.482769 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.482748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d30eed73-7ea8-4ead-9662-764a01687571-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.482874 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.482778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96rn\" (UniqueName: \"kubernetes.io/projected/d30eed73-7ea8-4ead-9662-764a01687571-kube-api-access-h96rn\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.482874 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.482800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d30eed73-7ea8-4ead-9662-764a01687571-metrics-client-ca\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.482874 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.482825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.482874 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.482868 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.483058 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.482893 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.483329 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.483309 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d30eed73-7ea8-4ead-9662-764a01687571-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.483570 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.483548 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d30eed73-7ea8-4ead-9662-764a01687571-metrics-client-ca\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.483662 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.483589 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.485385 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.485351 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.485467 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.485433 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d30eed73-7ea8-4ead-9662-764a01687571-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.529782 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.529743 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-n4jww"] Jul 02 16:58:17.534091 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.534065 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96rn\" (UniqueName: \"kubernetes.io/projected/d30eed73-7ea8-4ead-9662-764a01687571-kube-api-access-h96rn\") pod \"kube-state-metrics-845b954455-n4jww\" (UID: \"d30eed73-7ea8-4ead-9662-764a01687571\") " pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.677310 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.677280 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wdnp" Jul 02 16:58:17.678366 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.678348 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" Jul 02 16:58:17.878738 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.878693 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gcpm4"] Jul 02 16:58:17.881508 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.881492 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.886652 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.886622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-wtmp\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.886784 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.886661 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.886784 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.886715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-tls\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.886882 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.886820 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-root\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.886882 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.886863 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-textfile\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.887097 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.886923 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.887586 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.887379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facaaa7e-843e-427e-a094-fee3bfedd82f-metrics-client-ca\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.887586 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.887428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vbr\" (UniqueName: \"kubernetes.io/projected/facaaa7e-843e-427e-a094-fee3bfedd82f-kube-api-access-92vbr\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.887586 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.887500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-sys\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.890337 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.890199 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 16:58:17.898340 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.898321 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 16:58:17.898436 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.898381 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 16:58:17.899490 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.899470 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-n4jww"] Jul 02 16:58:17.900408 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:58:17.900385 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30eed73_7ea8_4ead_9662_764a01687571.slice/crio-0dfc3e8818f1736f116b38a457ccaa6ad1d93704096d5521dcb26baba61e64c7 WatchSource:0}: Error finding container 0dfc3e8818f1736f116b38a457ccaa6ad1d93704096d5521dcb26baba61e64c7: Status 404 returned error can't find the container with id 0dfc3e8818f1736f116b38a457ccaa6ad1d93704096d5521dcb26baba61e64c7 Jul 02 16:58:17.910303 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.910283 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wdnp"] Jul 02 16:58:17.915021 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:58:17.914996 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35992b57_3bd9_42c2_8287_c50dd2ddc6a9.slice/crio-b97920625927ae6cfa485025c29f8dc2f21c11e86648ecd86075ff027876975b WatchSource:0}: Error finding container b97920625927ae6cfa485025c29f8dc2f21c11e86648ecd86075ff027876975b: Status 404 returned error can't find the container with id b97920625927ae6cfa485025c29f8dc2f21c11e86648ecd86075ff027876975b Jul 02 16:58:17.934993 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.934948 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5m6cm\"" Jul 02 16:58:17.988808 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.988774 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.988808 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.988813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facaaa7e-843e-427e-a094-fee3bfedd82f-metrics-client-ca\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989012 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.988829 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vbr\" (UniqueName: \"kubernetes.io/projected/facaaa7e-843e-427e-a094-fee3bfedd82f-kube-api-access-92vbr\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989012 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.988862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-sys\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989012 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.988907 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-wtmp\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989012 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.988931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989012 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.988966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-tls\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989012 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.988998 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-sys\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989290 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.989047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-root\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989290 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.989080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-textfile\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989290 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.989105 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-wtmp\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989290 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.989147 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/facaaa7e-843e-427e-a094-fee3bfedd82f-root\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989425 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.989389 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989425 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.989408 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facaaa7e-843e-427e-a094-fee3bfedd82f-metrics-client-ca\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.989897 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.989875 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-textfile\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.991705 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.991685 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:17.991859 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:17.991841 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/facaaa7e-843e-427e-a094-fee3bfedd82f-node-exporter-tls\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:18.035812 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:18.035784 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vbr\" (UniqueName: \"kubernetes.io/projected/facaaa7e-843e-427e-a094-fee3bfedd82f-kube-api-access-92vbr\") pod \"node-exporter-gcpm4\" (UID: \"facaaa7e-843e-427e-a094-fee3bfedd82f\") " pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:18.193267 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:18.193192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gcpm4" Jul 02 16:58:18.203390 ip-10-0-134-244 kubenswrapper[2582]: W0702 16:58:18.203366 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacaaa7e_843e_427e_a094_fee3bfedd82f.slice/crio-99742f2ba36ac434c531ef287b49be7a54dbf7c5250d8fa952e3a60358b50b61 WatchSource:0}: Error finding container 99742f2ba36ac434c531ef287b49be7a54dbf7c5250d8fa952e3a60358b50b61: Status 404 returned error can't find the container with id 99742f2ba36ac434c531ef287b49be7a54dbf7c5250d8fa952e3a60358b50b61 Jul 02 16:58:18.792747 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:18.792696 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcpm4" event={"ID":"facaaa7e-843e-427e-a094-fee3bfedd82f","Type":"ContainerStarted","Data":"99742f2ba36ac434c531ef287b49be7a54dbf7c5250d8fa952e3a60358b50b61"} Jul 02 16:58:18.797504 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:18.797413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" event={"ID":"d30eed73-7ea8-4ead-9662-764a01687571","Type":"ContainerStarted","Data":"0dfc3e8818f1736f116b38a457ccaa6ad1d93704096d5521dcb26baba61e64c7"} Jul 02 16:58:18.800201 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:18.800130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wdnp" event={"ID":"35992b57-3bd9-42c2-8287-c50dd2ddc6a9","Type":"ContainerStarted","Data":"b97920625927ae6cfa485025c29f8dc2f21c11e86648ecd86075ff027876975b"} Jul 02 16:58:22.818712 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:22.818677 2582 generic.go:356] "Generic (PLEG): container finished" podID="facaaa7e-843e-427e-a094-fee3bfedd82f" containerID="6da951475be976ce6037e973d685d2fa1acd53519ceae1a33142f4fc0201328b" exitCode=0 Jul 02 16:58:22.819159 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:22.818771 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcpm4" event={"ID":"facaaa7e-843e-427e-a094-fee3bfedd82f","Type":"ContainerDied","Data":"6da951475be976ce6037e973d685d2fa1acd53519ceae1a33142f4fc0201328b"} Jul 02 16:58:22.821462 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:22.821424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" event={"ID":"d30eed73-7ea8-4ead-9662-764a01687571","Type":"ContainerStarted","Data":"0efd983ee642423ad47fcacbda3f41045c104d00345a7aca70acbca88030a5bc"} Jul 02 16:58:22.821582 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:22.821479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" event={"ID":"d30eed73-7ea8-4ead-9662-764a01687571","Type":"ContainerStarted","Data":"1054566a99c1c6a173eda2d742dcd613ae284a138d83080208f1c79cd6083c3e"} Jul 02 16:58:22.821582 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:22.821509 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" event={"ID":"d30eed73-7ea8-4ead-9662-764a01687571","Type":"ContainerStarted","Data":"36ae90c6196e7c87ec8140febf9711e00ce15912828d5bb52d9763cd44dcd797"} Jul 02 16:58:22.823103 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:22.823074 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wdnp" event={"ID":"35992b57-3bd9-42c2-8287-c50dd2ddc6a9","Type":"ContainerStarted","Data":"15d916f9e4649dcfce9cf2ae98cdb63576ebcb14f924593fbbb690f01a3a7c24"} Jul 02 16:58:22.932356 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:22.932312 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-n4jww" podStartSLOduration=1.783845433 podStartE2EDuration="5.932296274s" totalImagesPullingTime="4.148450841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:22.931460841 +0000 UTC m=+90.120777873" watchObservedRunningTime="2026-07-02 16:58:22.932296274 +0000 UTC m=+90.121613305" Jul 02 16:58:23.684855 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:23.684828 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:58:23.829127 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:23.829093 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wdnp" event={"ID":"35992b57-3bd9-42c2-8287-c50dd2ddc6a9","Type":"ContainerStarted","Data":"9865e3d0c1bfb7a30c19d07c298f219f938e41503d296709d0cc079963da9f06"} Jul 02 16:58:23.831677 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:23.831641 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcpm4" event={"ID":"facaaa7e-843e-427e-a094-fee3bfedd82f","Type":"ContainerStarted","Data":"d9714d394f036d878fc98cb311b8146371ac51a1510a016634aaadd983ecd54f"} Jul 02 16:58:23.831812 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:23.831706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcpm4" event={"ID":"facaaa7e-843e-427e-a094-fee3bfedd82f","Type":"ContainerStarted","Data":"4deb53e267feadbd70972caaac5ca2e17066b1a24b87ad1f589a240583056519"} Jul 02 16:58:23.866526 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:23.866466 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gcpm4" podStartSLOduration=2.925674845 podStartE2EDuration="6.86644719s" totalImagesPullingTime="3.8469301s" totalInitContainerRuntime="93.842245ms" isStatefulPod=true podCreationTimestamp="2026-07-02 16:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:23.865910782 +0000 UTC m=+91.055227817" watchObservedRunningTime="2026-07-02 16:58:23.86644719 +0000 UTC m=+91.055764223" Jul 02 16:58:25.843022 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:25.842931 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wdnp" event={"ID":"35992b57-3bd9-42c2-8287-c50dd2ddc6a9","Type":"ContainerStarted","Data":"ad199e3ec72563b24fa5d4915aa0c47d33f71a1a32571c1701d6c57c3ff0d91f"} Jul 02 16:58:25.889166 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:25.889120 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4wdnp" podStartSLOduration=9.549107683999999 podStartE2EDuration="16.889105191s" totalImagesPullingTime="7.339997507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:58:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:25.888320393 +0000 UTC m=+93.077637425" watchObservedRunningTime="2026-07-02 16:58:25.889105191 +0000 UTC m=+93.078422221" Jul 02 16:58:28.699556 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.699512 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" podUID="29934c88-bff5-40b7-a9d7-d63339318a6d" containerName="registry" containerID="cri-o://948d5176d5c192ed1fe3fc114439da1ed480150bfac45a36e59475aff7a50700" gracePeriod=30 Jul 02 16:58:28.859260 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.859229 2582 generic.go:356] "Generic (PLEG): container finished" podID="29934c88-bff5-40b7-a9d7-d63339318a6d" containerID="948d5176d5c192ed1fe3fc114439da1ed480150bfac45a36e59475aff7a50700" exitCode=0 Jul 02 16:58:28.859409 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.859262 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" event={"ID":"29934c88-bff5-40b7-a9d7-d63339318a6d","Type":"ContainerDied","Data":"948d5176d5c192ed1fe3fc114439da1ed480150bfac45a36e59475aff7a50700"} Jul 02 16:58:28.946313 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.946290 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:58:28.977400 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.977315 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-installation-pull-secrets\") pod \"29934c88-bff5-40b7-a9d7-d63339318a6d\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " Jul 02 16:58:28.977400 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.977358 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-trusted-ca\") pod \"29934c88-bff5-40b7-a9d7-d63339318a6d\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " Jul 02 16:58:28.977400 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.977385 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p77zg\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-kube-api-access-p77zg\") pod \"29934c88-bff5-40b7-a9d7-d63339318a6d\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " Jul 02 16:58:28.977656 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.977414 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-bound-sa-token\") pod \"29934c88-bff5-40b7-a9d7-d63339318a6d\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " Jul 02 16:58:28.977656 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.977440 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") pod \"29934c88-bff5-40b7-a9d7-d63339318a6d\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " Jul 02 16:58:28.977656 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.977473 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29934c88-bff5-40b7-a9d7-d63339318a6d-ca-trust-extracted\") pod \"29934c88-bff5-40b7-a9d7-d63339318a6d\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " Jul 02 16:58:28.977656 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.977519 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-image-registry-private-configuration\") pod \"29934c88-bff5-40b7-a9d7-d63339318a6d\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " Jul 02 16:58:28.977656 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.977560 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-certificates\") pod \"29934c88-bff5-40b7-a9d7-d63339318a6d\" (UID: \"29934c88-bff5-40b7-a9d7-d63339318a6d\") " Jul 02 16:58:28.978032 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.978003 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "29934c88-bff5-40b7-a9d7-d63339318a6d" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 16:58:28.978108 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.978089 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "29934c88-bff5-40b7-a9d7-d63339318a6d" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 16:58:28.980419 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.980372 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-kube-api-access-p77zg" (OuterVolumeSpecName: "kube-api-access-p77zg") pod "29934c88-bff5-40b7-a9d7-d63339318a6d" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d"). InnerVolumeSpecName "kube-api-access-p77zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 16:58:28.980553 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.980500 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "29934c88-bff5-40b7-a9d7-d63339318a6d" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 16:58:28.980858 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.980835 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "29934c88-bff5-40b7-a9d7-d63339318a6d" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 16:58:28.980940 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.980859 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "29934c88-bff5-40b7-a9d7-d63339318a6d" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 16:58:28.980940 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.980932 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "29934c88-bff5-40b7-a9d7-d63339318a6d" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 16:58:28.989517 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:28.989484 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29934c88-bff5-40b7-a9d7-d63339318a6d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "29934c88-bff5-40b7-a9d7-d63339318a6d" (UID: "29934c88-bff5-40b7-a9d7-d63339318a6d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 16:58:29.078366 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.078327 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-installation-pull-secrets\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 16:58:29.078366 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.078361 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-trusted-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 16:58:29.078366 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.078370 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p77zg\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-kube-api-access-p77zg\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 16:58:29.078583 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.078379 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-bound-sa-token\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 16:58:29.078583 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.078389 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 16:58:29.078583 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.078397 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29934c88-bff5-40b7-a9d7-d63339318a6d-ca-trust-extracted\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 16:58:29.078583 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.078406 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29934c88-bff5-40b7-a9d7-d63339318a6d-image-registry-private-configuration\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 16:58:29.078583 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.078415 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29934c88-bff5-40b7-a9d7-d63339318a6d-registry-certificates\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 16:58:29.864369 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.864329 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" Jul 02 16:58:29.864839 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.864328 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f77ff489c-7d5tp" event={"ID":"29934c88-bff5-40b7-a9d7-d63339318a6d","Type":"ContainerDied","Data":"f131e402e892e0ccb3c0350c9ae1682aac4b5f1fb28ac88ad4bd189412d06486"} Jul 02 16:58:29.864839 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.864470 2582 scope.go:117] "RemoveContainer" containerID="948d5176d5c192ed1fe3fc114439da1ed480150bfac45a36e59475aff7a50700" Jul 02 16:58:29.888970 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.888946 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f77ff489c-7d5tp"] Jul 02 16:58:29.893327 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:29.893306 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f77ff489c-7d5tp"] Jul 02 16:58:31.264083 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:31.264049 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29934c88-bff5-40b7-a9d7-d63339318a6d" path="/var/lib/kubelet/pods/29934c88-bff5-40b7-a9d7-d63339318a6d/volumes" Jul 02 16:58:36.741400 ip-10-0-134-244 kubenswrapper[2582]: I0702 16:58:36.741370 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g6mlh" Jul 02 17:01:53.190774 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:01:53.190742 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:01:53.195311 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:01:53.195290 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 17:01:53.196146 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:01:53.196124 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:02:58.898161 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.898075 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq"] Jul 02 17:02:58.898555 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.898457 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29934c88-bff5-40b7-a9d7-d63339318a6d" containerName="registry" Jul 02 17:02:58.898555 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.898475 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="29934c88-bff5-40b7-a9d7-d63339318a6d" containerName="registry" Jul 02 17:02:58.898555 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.898552 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="29934c88-bff5-40b7-a9d7-d63339318a6d" containerName="registry" Jul 02 17:02:58.901491 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.901472 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" Jul 02 17:02:58.904249 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.904231 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jmjdq\"" Jul 02 17:02:58.904340 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.904270 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 02 17:02:58.904656 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.904641 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 02 17:02:58.917510 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.917486 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq"] Jul 02 17:02:58.970577 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.970542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79dq\" (UniqueName: \"kubernetes.io/projected/7c47dd99-d8a8-4cb7-bb35-4c717b6613b8-kube-api-access-v79dq\") pod \"cert-manager-operator-controller-manager-55c444bf86-t4nqq\" (UID: \"7c47dd99-d8a8-4cb7-bb35-4c717b6613b8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" Jul 02 17:02:58.970577 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:58.970580 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c47dd99-d8a8-4cb7-bb35-4c717b6613b8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t4nqq\" (UID: \"7c47dd99-d8a8-4cb7-bb35-4c717b6613b8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" Jul 02 17:02:59.071510 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:59.071477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c47dd99-d8a8-4cb7-bb35-4c717b6613b8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t4nqq\" (UID: \"7c47dd99-d8a8-4cb7-bb35-4c717b6613b8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" Jul 02 17:02:59.071663 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:59.071541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79dq\" (UniqueName: \"kubernetes.io/projected/7c47dd99-d8a8-4cb7-bb35-4c717b6613b8-kube-api-access-v79dq\") pod \"cert-manager-operator-controller-manager-55c444bf86-t4nqq\" (UID: \"7c47dd99-d8a8-4cb7-bb35-4c717b6613b8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" Jul 02 17:02:59.071869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:59.071853 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c47dd99-d8a8-4cb7-bb35-4c717b6613b8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t4nqq\" (UID: \"7c47dd99-d8a8-4cb7-bb35-4c717b6613b8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" Jul 02 17:02:59.080600 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:59.080575 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79dq\" (UniqueName: \"kubernetes.io/projected/7c47dd99-d8a8-4cb7-bb35-4c717b6613b8-kube-api-access-v79dq\") pod \"cert-manager-operator-controller-manager-55c444bf86-t4nqq\" (UID: \"7c47dd99-d8a8-4cb7-bb35-4c717b6613b8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" Jul 02 17:02:59.210780 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:59.210676 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" Jul 02 17:02:59.359142 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:59.359117 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq"] Jul 02 17:02:59.361817 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:02:59.361790 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c47dd99_d8a8_4cb7_bb35_4c717b6613b8.slice/crio-b26962ccd161644309a1af56e7d04d6cc472f4f41ab387c64daa9bf7db6ac2df WatchSource:0}: Error finding container b26962ccd161644309a1af56e7d04d6cc472f4f41ab387c64daa9bf7db6ac2df: Status 404 returned error can't find the container with id b26962ccd161644309a1af56e7d04d6cc472f4f41ab387c64daa9bf7db6ac2df Jul 02 17:02:59.364285 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:59.364269 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 17:02:59.728228 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:02:59.728196 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" event={"ID":"7c47dd99-d8a8-4cb7-bb35-4c717b6613b8","Type":"ContainerStarted","Data":"b26962ccd161644309a1af56e7d04d6cc472f4f41ab387c64daa9bf7db6ac2df"} Jul 02 17:03:02.740195 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:02.740152 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" event={"ID":"7c47dd99-d8a8-4cb7-bb35-4c717b6613b8","Type":"ContainerStarted","Data":"0ec344ff0d94bb6ec133c924f7ea69ab3c345df89ef0bfe732a1c583ba7f3727"} Jul 02 17:03:02.763423 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:02.763374 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t4nqq" podStartSLOduration=2.3429168320000002 podStartE2EDuration="4.76335881s" totalImagesPullingTime="2.420441978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:03:02.761711966 +0000 UTC m=+369.951028989" watchObservedRunningTime="2026-07-02 17:03:02.76335881 +0000 UTC m=+369.952675839" Jul 02 17:03:05.397483 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.397451 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt"] Jul 02 17:03:05.400809 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.400794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" Jul 02 17:03:05.406056 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.403093 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-djdrs\"" Jul 02 17:03:05.406056 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.403240 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 17:03:05.406056 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.404064 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 17:03:05.411531 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.411497 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt"] Jul 02 17:03:05.520871 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.520828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef192af7-8159-4554-a799-1a481079b1fe-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7q7wt\" (UID: \"ef192af7-8159-4554-a799-1a481079b1fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" Jul 02 17:03:05.521046 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.520880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg4vv\" (UniqueName: \"kubernetes.io/projected/ef192af7-8159-4554-a799-1a481079b1fe-kube-api-access-hg4vv\") pod \"cert-manager-cainjector-559cdd8c87-7q7wt\" (UID: \"ef192af7-8159-4554-a799-1a481079b1fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" Jul 02 17:03:05.621851 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.621811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef192af7-8159-4554-a799-1a481079b1fe-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7q7wt\" (UID: \"ef192af7-8159-4554-a799-1a481079b1fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" Jul 02 17:03:05.622023 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.621863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg4vv\" (UniqueName: \"kubernetes.io/projected/ef192af7-8159-4554-a799-1a481079b1fe-kube-api-access-hg4vv\") pod \"cert-manager-cainjector-559cdd8c87-7q7wt\" (UID: \"ef192af7-8159-4554-a799-1a481079b1fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" Jul 02 17:03:05.630614 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.630579 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg4vv\" (UniqueName: \"kubernetes.io/projected/ef192af7-8159-4554-a799-1a481079b1fe-kube-api-access-hg4vv\") pod \"cert-manager-cainjector-559cdd8c87-7q7wt\" (UID: \"ef192af7-8159-4554-a799-1a481079b1fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" Jul 02 17:03:05.630614 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.630609 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef192af7-8159-4554-a799-1a481079b1fe-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7q7wt\" (UID: \"ef192af7-8159-4554-a799-1a481079b1fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" Jul 02 17:03:05.713479 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.713385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" Jul 02 17:03:05.837479 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:05.837386 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt"] Jul 02 17:03:05.839951 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:03:05.839920 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef192af7_8159_4554_a799_1a481079b1fe.slice/crio-53ba8b3fc0bda89db5a1602c9d122843a2c99517bffcc44e998ae936a29c85c4 WatchSource:0}: Error finding container 53ba8b3fc0bda89db5a1602c9d122843a2c99517bffcc44e998ae936a29c85c4: Status 404 returned error can't find the container with id 53ba8b3fc0bda89db5a1602c9d122843a2c99517bffcc44e998ae936a29c85c4 Jul 02 17:03:06.757486 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:06.757444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" event={"ID":"ef192af7-8159-4554-a799-1a481079b1fe","Type":"ContainerStarted","Data":"53ba8b3fc0bda89db5a1602c9d122843a2c99517bffcc44e998ae936a29c85c4"} Jul 02 17:03:08.766400 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:08.766364 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" event={"ID":"ef192af7-8159-4554-a799-1a481079b1fe","Type":"ContainerStarted","Data":"39a65f5c001cd0dc71dfcfe7a4f901dc871968113f1baec15a9e709c8527dcf9"} Jul 02 17:03:08.792893 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:08.792841 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7q7wt" podStartSLOduration=1.390288304 podStartE2EDuration="3.79282335s" totalImagesPullingTime="2.402535046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:03:08.791871678 +0000 UTC m=+375.981188710" watchObservedRunningTime="2026-07-02 17:03:08.79282335 +0000 UTC m=+375.982140380" Jul 02 17:03:16.184345 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.184313 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb"] Jul 02 17:03:16.187244 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.187221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" Jul 02 17:03:16.191840 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.191819 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cvm7t\"" Jul 02 17:03:16.192935 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.192709 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 17:03:16.192935 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.192924 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 17:03:16.196124 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.196104 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb"] Jul 02 17:03:16.202681 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.202651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f3e190-ba80-4420-97c2-0fe12039bf0b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w4qdb\" (UID: \"10f3e190-ba80-4420-97c2-0fe12039bf0b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" Jul 02 17:03:16.202840 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.202695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpkhb\" (UniqueName: \"kubernetes.io/projected/10f3e190-ba80-4420-97c2-0fe12039bf0b-kube-api-access-bpkhb\") pod \"openshift-lws-operator-fd8ccff4c-w4qdb\" (UID: \"10f3e190-ba80-4420-97c2-0fe12039bf0b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" Jul 02 17:03:16.303549 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.303508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f3e190-ba80-4420-97c2-0fe12039bf0b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w4qdb\" (UID: \"10f3e190-ba80-4420-97c2-0fe12039bf0b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" Jul 02 17:03:16.303549 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.303554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpkhb\" (UniqueName: \"kubernetes.io/projected/10f3e190-ba80-4420-97c2-0fe12039bf0b-kube-api-access-bpkhb\") pod \"openshift-lws-operator-fd8ccff4c-w4qdb\" (UID: \"10f3e190-ba80-4420-97c2-0fe12039bf0b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" Jul 02 17:03:16.303963 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.303942 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f3e190-ba80-4420-97c2-0fe12039bf0b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w4qdb\" (UID: \"10f3e190-ba80-4420-97c2-0fe12039bf0b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" Jul 02 17:03:16.311478 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.311451 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpkhb\" (UniqueName: \"kubernetes.io/projected/10f3e190-ba80-4420-97c2-0fe12039bf0b-kube-api-access-bpkhb\") pod \"openshift-lws-operator-fd8ccff4c-w4qdb\" (UID: \"10f3e190-ba80-4420-97c2-0fe12039bf0b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" Jul 02 17:03:16.503173 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.503069 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" Jul 02 17:03:16.629005 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.628971 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb"] Jul 02 17:03:16.631993 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:03:16.631966 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10f3e190_ba80_4420_97c2_0fe12039bf0b.slice/crio-242ecb470227ea7588ca0aa54fb7eda9a0d344673b34303893110be70ca41f06 WatchSource:0}: Error finding container 242ecb470227ea7588ca0aa54fb7eda9a0d344673b34303893110be70ca41f06: Status 404 returned error can't find the container with id 242ecb470227ea7588ca0aa54fb7eda9a0d344673b34303893110be70ca41f06 Jul 02 17:03:16.795558 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:16.795482 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" event={"ID":"10f3e190-ba80-4420-97c2-0fe12039bf0b","Type":"ContainerStarted","Data":"242ecb470227ea7588ca0aa54fb7eda9a0d344673b34303893110be70ca41f06"} Jul 02 17:03:19.811806 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:19.811772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" event={"ID":"10f3e190-ba80-4420-97c2-0fe12039bf0b","Type":"ContainerStarted","Data":"22ad6e4f4690fde9bb597363dc5cc3ed0a98bfc1078aab69f6ef369398c766fc"} Jul 02 17:03:19.829809 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:19.829755 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w4qdb" podStartSLOduration=1.35873446 podStartE2EDuration="3.829739576s" totalImagesPullingTime="2.471005116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:03:19.82880161 +0000 UTC m=+387.018118642" watchObservedRunningTime="2026-07-02 17:03:19.829739576 +0000 UTC m=+387.019056598" Jul 02 17:03:35.768059 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.768021 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825"] Jul 02 17:03:35.773800 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.773772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.779263 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.779233 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tl59g\"" Jul 02 17:03:35.779418 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.779298 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 17:03:35.779487 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.779466 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 17:03:35.779576 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.779559 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 17:03:35.779656 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.779599 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 17:03:35.790539 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.790517 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825"] Jul 02 17:03:35.838532 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.838503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/adb297bd-54ae-429d-97ed-061bad4f5635-manifests\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.838664 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.838536 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adb297bd-54ae-429d-97ed-061bad4f5635-tmp\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.838664 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.838564 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b466l\" (UniqueName: \"kubernetes.io/projected/adb297bd-54ae-429d-97ed-061bad4f5635-kube-api-access-b466l\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.838664 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.838604 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adb297bd-54ae-429d-97ed-061bad4f5635-webhook-cert\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.838664 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.838641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/adb297bd-54ae-429d-97ed-061bad4f5635-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.939368 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.939333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/adb297bd-54ae-429d-97ed-061bad4f5635-manifests\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.939368 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.939372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adb297bd-54ae-429d-97ed-061bad4f5635-tmp\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.939594 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.939407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b466l\" (UniqueName: \"kubernetes.io/projected/adb297bd-54ae-429d-97ed-061bad4f5635-kube-api-access-b466l\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.939594 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.939430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adb297bd-54ae-429d-97ed-061bad4f5635-webhook-cert\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.939594 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.939450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/adb297bd-54ae-429d-97ed-061bad4f5635-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.939841 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.939814 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/adb297bd-54ae-429d-97ed-061bad4f5635-manifests\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.941869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.941846 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adb297bd-54ae-429d-97ed-061bad4f5635-tmp\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.942075 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.942049 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adb297bd-54ae-429d-97ed-061bad4f5635-webhook-cert\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.942168 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.942153 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/adb297bd-54ae-429d-97ed-061bad4f5635-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:35.951888 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:35.951863 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b466l\" (UniqueName: \"kubernetes.io/projected/adb297bd-54ae-429d-97ed-061bad4f5635-kube-api-access-b466l\") pod \"opendatahub-operator-controller-manager-8b6c9f55f-82825\" (UID: \"adb297bd-54ae-429d-97ed-061bad4f5635\") " pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:36.090898 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:36.090814 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:36.221695 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:36.221670 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825"] Jul 02 17:03:36.224025 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:03:36.224003 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb297bd_54ae_429d_97ed_061bad4f5635.slice/crio-96d7b25ef1aa67636b7d0b50886c893b9104d16c942728d6b079e8db68525eec WatchSource:0}: Error finding container 96d7b25ef1aa67636b7d0b50886c893b9104d16c942728d6b079e8db68525eec: Status 404 returned error can't find the container with id 96d7b25ef1aa67636b7d0b50886c893b9104d16c942728d6b079e8db68525eec Jul 02 17:03:36.881068 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:36.881023 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" event={"ID":"adb297bd-54ae-429d-97ed-061bad4f5635","Type":"ContainerStarted","Data":"96d7b25ef1aa67636b7d0b50886c893b9104d16c942728d6b079e8db68525eec"} Jul 02 17:03:39.895854 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:39.895822 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" event={"ID":"adb297bd-54ae-429d-97ed-061bad4f5635","Type":"ContainerStarted","Data":"6a19845f3cc2f337fc054d616caad39af14091570d14e4f8b9f48f62f08023f9"} Jul 02 17:03:40.901179 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:40.901144 2582 generic.go:356] "Generic (PLEG): container finished" podID="adb297bd-54ae-429d-97ed-061bad4f5635" containerID="6a19845f3cc2f337fc054d616caad39af14091570d14e4f8b9f48f62f08023f9" exitCode=0 Jul 02 17:03:40.901534 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:40.901232 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" event={"ID":"adb297bd-54ae-429d-97ed-061bad4f5635","Type":"ContainerDied","Data":"6a19845f3cc2f337fc054d616caad39af14091570d14e4f8b9f48f62f08023f9"} Jul 02 17:03:41.911592 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:41.911551 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" event={"ID":"adb297bd-54ae-429d-97ed-061bad4f5635","Type":"ContainerStarted","Data":"b0b449b45010d8d3a386d16c4d769ea5cd10945ffb2786ef88a368d941be27ae"} Jul 02 17:03:41.911999 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:41.911642 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:41.936888 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:41.936838 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" podStartSLOduration=2.570826409 podStartE2EDuration="6.936819022s" totalImagesPullingTime="3.235927566s" totalInitContainerRuntime="1.130065047s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:03:41.93600796 +0000 UTC m=+409.125324991" watchObservedRunningTime="2026-07-02 17:03:41.936819022 +0000 UTC m=+409.126136057" Jul 02 17:03:52.919878 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:52.919847 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8b6c9f55f-82825" Jul 02 17:03:56.072484 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.072451 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp"] Jul 02 17:03:56.077697 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.077668 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.081930 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.081909 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 17:03:56.082032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.081944 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 17:03:56.082032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.081944 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 17:03:56.084232 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.084208 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 02 17:03:56.088633 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.088617 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f2kds\"" Jul 02 17:03:56.106031 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.106006 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp"] Jul 02 17:03:56.188448 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.188412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-tmp\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.188448 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.188447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd4z7\" (UniqueName: \"kubernetes.io/projected/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-kube-api-access-qd4z7\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.188630 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.188512 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-tls-certs\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.249299 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.249270 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qngj2"] Jul 02 17:03:56.251377 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.251363 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:56.256409 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.256384 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 02 17:03:56.256409 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.256395 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9m85q\"" Jul 02 17:03:56.289713 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.289690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/869652f0-e49a-4195-8cbf-7367b9599b38-cert\") pod \"odh-model-controller-7998c67dc4-qngj2\" (UID: \"869652f0-e49a-4195-8cbf-7367b9599b38\") " pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:56.289817 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.289745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-tmp\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.289817 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.289766 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd4z7\" (UniqueName: \"kubernetes.io/projected/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-kube-api-access-qd4z7\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.289817 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.289802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-tls-certs\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.289932 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.289821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfpxg\" (UniqueName: \"kubernetes.io/projected/869652f0-e49a-4195-8cbf-7367b9599b38-kube-api-access-cfpxg\") pod \"odh-model-controller-7998c67dc4-qngj2\" (UID: \"869652f0-e49a-4195-8cbf-7367b9599b38\") " pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:56.292169 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.292148 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-tmp\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.292275 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.292200 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-tls-certs\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.314769 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.314739 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd4z7\" (UniqueName: \"kubernetes.io/projected/44b6c6d7-667b-4faa-b1e6-4a033e0a0032-kube-api-access-qd4z7\") pod \"kube-auth-proxy-7c5785f89f-fqhjp\" (UID: \"44b6c6d7-667b-4faa-b1e6-4a033e0a0032\") " pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.353272 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.353217 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qngj2"] Jul 02 17:03:56.388222 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.388197 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" Jul 02 17:03:56.390259 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.390106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/869652f0-e49a-4195-8cbf-7367b9599b38-cert\") pod \"odh-model-controller-7998c67dc4-qngj2\" (UID: \"869652f0-e49a-4195-8cbf-7367b9599b38\") " pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:56.390259 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.390170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfpxg\" (UniqueName: \"kubernetes.io/projected/869652f0-e49a-4195-8cbf-7367b9599b38-kube-api-access-cfpxg\") pod \"odh-model-controller-7998c67dc4-qngj2\" (UID: \"869652f0-e49a-4195-8cbf-7367b9599b38\") " pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:56.390339 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:03:56.390254 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 02 17:03:56.390339 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:03:56.390312 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869652f0-e49a-4195-8cbf-7367b9599b38-cert podName:869652f0-e49a-4195-8cbf-7367b9599b38 nodeName:}" failed. No retries permitted until 2026-07-02 17:03:56.890294353 +0000 UTC m=+424.079611365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/869652f0-e49a-4195-8cbf-7367b9599b38-cert") pod "odh-model-controller-7998c67dc4-qngj2" (UID: "869652f0-e49a-4195-8cbf-7367b9599b38") : secret "odh-model-controller-webhook-cert" not found Jul 02 17:03:56.428979 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.428947 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfpxg\" (UniqueName: \"kubernetes.io/projected/869652f0-e49a-4195-8cbf-7367b9599b38-kube-api-access-cfpxg\") pod \"odh-model-controller-7998c67dc4-qngj2\" (UID: \"869652f0-e49a-4195-8cbf-7367b9599b38\") " pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:56.549082 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.549053 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp"] Jul 02 17:03:56.552374 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:03:56.552347 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b6c6d7_667b_4faa_b1e6_4a033e0a0032.slice/crio-4e560289bbc452a2f3209b43ceb90fa325dd7fd77548180475036973f09d34c3 WatchSource:0}: Error finding container 4e560289bbc452a2f3209b43ceb90fa325dd7fd77548180475036973f09d34c3: Status 404 returned error can't find the container with id 4e560289bbc452a2f3209b43ceb90fa325dd7fd77548180475036973f09d34c3 Jul 02 17:03:56.893434 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.893401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/869652f0-e49a-4195-8cbf-7367b9599b38-cert\") pod \"odh-model-controller-7998c67dc4-qngj2\" (UID: \"869652f0-e49a-4195-8cbf-7367b9599b38\") " pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:56.895745 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.895706 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/869652f0-e49a-4195-8cbf-7367b9599b38-cert\") pod \"odh-model-controller-7998c67dc4-qngj2\" (UID: \"869652f0-e49a-4195-8cbf-7367b9599b38\") " pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:56.975849 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:56.975819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" event={"ID":"44b6c6d7-667b-4faa-b1e6-4a033e0a0032","Type":"ContainerStarted","Data":"4e560289bbc452a2f3209b43ceb90fa325dd7fd77548180475036973f09d34c3"} Jul 02 17:03:57.160576 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:57.160494 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:03:57.309525 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:57.309492 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qngj2"] Jul 02 17:03:57.312326 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:03:57.312295 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869652f0_e49a_4195_8cbf_7367b9599b38.slice/crio-c411b2d6501a272120607191fed8a05447b7e285d997a1bc262573b00f418f52 WatchSource:0}: Error finding container c411b2d6501a272120607191fed8a05447b7e285d997a1bc262573b00f418f52: Status 404 returned error can't find the container with id c411b2d6501a272120607191fed8a05447b7e285d997a1bc262573b00f418f52 Jul 02 17:03:57.924437 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:57.923101 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zd7v5"] Jul 02 17:03:57.925953 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:57.925922 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:03:57.929674 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:57.929649 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 02 17:03:57.982540 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:57.982501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" event={"ID":"869652f0-e49a-4195-8cbf-7367b9599b38","Type":"ContainerStarted","Data":"c411b2d6501a272120607191fed8a05447b7e285d997a1bc262573b00f418f52"} Jul 02 17:03:58.004653 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.004609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpw4s\" (UniqueName: \"kubernetes.io/projected/37f119d3-97a0-476f-bb44-31e80a93d991-kube-api-access-gpw4s\") pod \"model-serving-api-54d89bd5fc-zd7v5\" (UID: \"37f119d3-97a0-476f-bb44-31e80a93d991\") " pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:03:58.004653 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.004667 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f119d3-97a0-476f-bb44-31e80a93d991-tls-certs\") pod \"model-serving-api-54d89bd5fc-zd7v5\" (UID: \"37f119d3-97a0-476f-bb44-31e80a93d991\") " pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:03:58.080188 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.080113 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zd7v5"] Jul 02 17:03:58.105500 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.105451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpw4s\" (UniqueName: \"kubernetes.io/projected/37f119d3-97a0-476f-bb44-31e80a93d991-kube-api-access-gpw4s\") pod \"model-serving-api-54d89bd5fc-zd7v5\" (UID: \"37f119d3-97a0-476f-bb44-31e80a93d991\") " pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:03:58.105669 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.105506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f119d3-97a0-476f-bb44-31e80a93d991-tls-certs\") pod \"model-serving-api-54d89bd5fc-zd7v5\" (UID: \"37f119d3-97a0-476f-bb44-31e80a93d991\") " pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:03:58.108704 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.108676 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f119d3-97a0-476f-bb44-31e80a93d991-tls-certs\") pod \"model-serving-api-54d89bd5fc-zd7v5\" (UID: \"37f119d3-97a0-476f-bb44-31e80a93d991\") " pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:03:58.168760 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.168693 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpw4s\" (UniqueName: \"kubernetes.io/projected/37f119d3-97a0-476f-bb44-31e80a93d991-kube-api-access-gpw4s\") pod \"model-serving-api-54d89bd5fc-zd7v5\" (UID: \"37f119d3-97a0-476f-bb44-31e80a93d991\") " pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:03:58.241568 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.241477 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:03:58.447246 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:58.447216 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zd7v5"] Jul 02 17:03:59.082646 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:03:59.082617 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f119d3_97a0_476f_bb44_31e80a93d991.slice/crio-9b4d44619672c1add90dc9757c8133949ae952f332c331b62e7b00016fbf10d9 WatchSource:0}: Error finding container 9b4d44619672c1add90dc9757c8133949ae952f332c331b62e7b00016fbf10d9: Status 404 returned error can't find the container with id 9b4d44619672c1add90dc9757c8133949ae952f332c331b62e7b00016fbf10d9 Jul 02 17:03:59.993813 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:03:59.993764 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" event={"ID":"37f119d3-97a0-476f-bb44-31e80a93d991","Type":"ContainerStarted","Data":"9b4d44619672c1add90dc9757c8133949ae952f332c331b62e7b00016fbf10d9"} Jul 02 17:04:01.001642 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:01.001602 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" event={"ID":"44b6c6d7-667b-4faa-b1e6-4a033e0a0032","Type":"ContainerStarted","Data":"fc0e5b6653620af985fc6766ea67a84a621922050b87a6ddb39a38d1c578e635"} Jul 02 17:04:01.002993 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:01.002964 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" event={"ID":"869652f0-e49a-4195-8cbf-7367b9599b38","Type":"ContainerStarted","Data":"43217b10ebdf2e579c773bfbf1cc3802eca073130eec7e7bc9c9ec70771dc4ac"} Jul 02 17:04:01.003111 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:01.003094 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:04:01.034133 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:01.034094 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c5785f89f-fqhjp" podStartSLOduration=1.216613786 podStartE2EDuration="5.034079041s" totalImagesPullingTime="3.817465255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:04:01.033058258 +0000 UTC m=+428.222375291" watchObservedRunningTime="2026-07-02 17:04:01.034079041 +0000 UTC m=+428.223396072" Jul 02 17:04:01.054210 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:01.054169 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" podStartSLOduration=1.9492669230000002 podStartE2EDuration="5.054148384s" totalImagesPullingTime="3.104881461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:04:01.052789017 +0000 UTC m=+428.242106049" watchObservedRunningTime="2026-07-02 17:04:01.054148384 +0000 UTC m=+428.243465420" Jul 02 17:04:02.009559 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.009517 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" event={"ID":"37f119d3-97a0-476f-bb44-31e80a93d991","Type":"ContainerStarted","Data":"099011724a9d00f043f1990dbe0b5014749cba54ea76e1a92b6799879ffc2738"} Jul 02 17:04:02.045226 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.045156 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" podStartSLOduration=3.110320437 podStartE2EDuration="5.045136501s" totalImagesPullingTime="1.934816064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:04:02.044197142 +0000 UTC m=+429.233514173" watchObservedRunningTime="2026-07-02 17:04:02.045136501 +0000 UTC m=+429.234453535" Jul 02 17:04:02.346581 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.346484 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff"] Jul 02 17:04:02.350168 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.350142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.354768 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.354735 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 17:04:02.354768 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.354750 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8bdb2\"" Jul 02 17:04:02.354945 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.354788 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 17:04:02.354945 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.354827 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 17:04:02.366304 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.366277 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff"] Jul 02 17:04:02.448579 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.448542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6frz\" (UniqueName: \"kubernetes.io/projected/7825283b-118e-4100-ba7e-07ea67e51d1f-kube-api-access-b6frz\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.448777 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.448597 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7825283b-118e-4100-ba7e-07ea67e51d1f-manager-config\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.448777 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.448691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7825283b-118e-4100-ba7e-07ea67e51d1f-metrics-cert\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.448866 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.448776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7825283b-118e-4100-ba7e-07ea67e51d1f-cert\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.549123 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.549089 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7825283b-118e-4100-ba7e-07ea67e51d1f-cert\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.549123 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.549129 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6frz\" (UniqueName: \"kubernetes.io/projected/7825283b-118e-4100-ba7e-07ea67e51d1f-kube-api-access-b6frz\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.549360 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.549165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7825283b-118e-4100-ba7e-07ea67e51d1f-manager-config\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.549360 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.549201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7825283b-118e-4100-ba7e-07ea67e51d1f-metrics-cert\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.550011 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.549982 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7825283b-118e-4100-ba7e-07ea67e51d1f-manager-config\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.551766 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.551742 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7825283b-118e-4100-ba7e-07ea67e51d1f-metrics-cert\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.551873 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.551744 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7825283b-118e-4100-ba7e-07ea67e51d1f-cert\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.566096 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.566003 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6frz\" (UniqueName: \"kubernetes.io/projected/7825283b-118e-4100-ba7e-07ea67e51d1f-kube-api-access-b6frz\") pod \"lws-controller-manager-6f677bc8d7-5cvff\" (UID: \"7825283b-118e-4100-ba7e-07ea67e51d1f\") " pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.658766 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.658708 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:02.797066 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:02.797041 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff"] Jul 02 17:04:02.799517 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:04:02.799486 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7825283b_118e_4100_ba7e_07ea67e51d1f.slice/crio-671a8ac6d9006d57afa8104c3e5cc5bb5105da5555c5ad6840a1fbb6f7d9a97d WatchSource:0}: Error finding container 671a8ac6d9006d57afa8104c3e5cc5bb5105da5555c5ad6840a1fbb6f7d9a97d: Status 404 returned error can't find the container with id 671a8ac6d9006d57afa8104c3e5cc5bb5105da5555c5ad6840a1fbb6f7d9a97d Jul 02 17:04:03.014823 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:03.014716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" event={"ID":"7825283b-118e-4100-ba7e-07ea67e51d1f","Type":"ContainerStarted","Data":"671a8ac6d9006d57afa8104c3e5cc5bb5105da5555c5ad6840a1fbb6f7d9a97d"} Jul 02 17:04:03.015169 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:03.014869 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:04:05.024521 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:05.024489 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" event={"ID":"7825283b-118e-4100-ba7e-07ea67e51d1f","Type":"ContainerStarted","Data":"ccbd33638f8c9f23685d0ecee309c07f36b4264f7039619c2c501c068d1a5e9b"} Jul 02 17:04:05.024933 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:05.024592 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:04:05.121327 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:05.121276 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" podStartSLOduration=1.444887132 podStartE2EDuration="3.121259324s" totalImagesPullingTime="1.676372192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:04:05.118995705 +0000 UTC m=+432.308312733" watchObservedRunningTime="2026-07-02 17:04:05.121259324 +0000 UTC m=+432.310576356" Jul 02 17:04:12.012354 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:12.012325 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qngj2" Jul 02 17:04:14.024026 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:14.023997 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zd7v5" Jul 02 17:04:16.031549 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:04:16.031470 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f677bc8d7-5cvff" Jul 02 17:06:26.583222 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.583190 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-kmzwk"] Jul 02 17:06:26.585337 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.585321 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kmzwk" Jul 02 17:06:26.587869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.587843 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ztk5p\"" Jul 02 17:06:26.587996 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.587944 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 17:06:26.588487 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.588471 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 17:06:26.595035 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.595009 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmzwk"] Jul 02 17:06:26.689393 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.689361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtngc\" (UniqueName: \"kubernetes.io/projected/f25b1961-e178-4c5e-ab99-55eaf9155a51-kube-api-access-mtngc\") pod \"authorino-58cd955675-kmzwk\" (UID: \"f25b1961-e178-4c5e-ab99-55eaf9155a51\") " pod="kuadrant-system/authorino-58cd955675-kmzwk" Jul 02 17:06:26.790079 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.790043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtngc\" (UniqueName: \"kubernetes.io/projected/f25b1961-e178-4c5e-ab99-55eaf9155a51-kube-api-access-mtngc\") pod \"authorino-58cd955675-kmzwk\" (UID: \"f25b1961-e178-4c5e-ab99-55eaf9155a51\") " pod="kuadrant-system/authorino-58cd955675-kmzwk" Jul 02 17:06:26.792198 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.792174 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-5jt4l"] Jul 02 17:06:26.794324 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.794308 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5jt4l" Jul 02 17:06:26.799934 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.799912 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5jt4l"] Jul 02 17:06:26.802340 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.802316 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtngc\" (UniqueName: \"kubernetes.io/projected/f25b1961-e178-4c5e-ab99-55eaf9155a51-kube-api-access-mtngc\") pod \"authorino-58cd955675-kmzwk\" (UID: \"f25b1961-e178-4c5e-ab99-55eaf9155a51\") " pod="kuadrant-system/authorino-58cd955675-kmzwk" Jul 02 17:06:26.891189 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.891162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654xp\" (UniqueName: \"kubernetes.io/projected/5dbd25ed-304c-4aac-9954-b9aa9b7f2f99-kube-api-access-654xp\") pod \"authorino-75d556bd77-5jt4l\" (UID: \"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99\") " pod="kuadrant-system/authorino-75d556bd77-5jt4l" Jul 02 17:06:26.895097 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.895078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kmzwk" Jul 02 17:06:26.992445 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:26.992407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654xp\" (UniqueName: \"kubernetes.io/projected/5dbd25ed-304c-4aac-9954-b9aa9b7f2f99-kube-api-access-654xp\") pod \"authorino-75d556bd77-5jt4l\" (UID: \"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99\") " pod="kuadrant-system/authorino-75d556bd77-5jt4l" Jul 02 17:06:27.001269 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:27.001244 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654xp\" (UniqueName: \"kubernetes.io/projected/5dbd25ed-304c-4aac-9954-b9aa9b7f2f99-kube-api-access-654xp\") pod \"authorino-75d556bd77-5jt4l\" (UID: \"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99\") " pod="kuadrant-system/authorino-75d556bd77-5jt4l" Jul 02 17:06:27.018690 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:27.018663 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmzwk"] Jul 02 17:06:27.022199 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:06:27.022173 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf25b1961_e178_4c5e_ab99_55eaf9155a51.slice/crio-d92917166092973fec338e895c7678baea10d0b3d818b447329244d42e33b137 WatchSource:0}: Error finding container d92917166092973fec338e895c7678baea10d0b3d818b447329244d42e33b137: Status 404 returned error can't find the container with id d92917166092973fec338e895c7678baea10d0b3d818b447329244d42e33b137 Jul 02 17:06:27.103848 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:27.103818 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5jt4l" Jul 02 17:06:27.223917 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:27.223761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5jt4l"] Jul 02 17:06:27.226492 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:06:27.226464 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dbd25ed_304c_4aac_9954_b9aa9b7f2f99.slice/crio-647ae7bb15dd32abf548ade4e3fa9d9c4ad76c4d0756e075a6b77f6a94dda41d WatchSource:0}: Error finding container 647ae7bb15dd32abf548ade4e3fa9d9c4ad76c4d0756e075a6b77f6a94dda41d: Status 404 returned error can't find the container with id 647ae7bb15dd32abf548ade4e3fa9d9c4ad76c4d0756e075a6b77f6a94dda41d Jul 02 17:06:27.607354 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:27.607272 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kmzwk" event={"ID":"f25b1961-e178-4c5e-ab99-55eaf9155a51","Type":"ContainerStarted","Data":"d92917166092973fec338e895c7678baea10d0b3d818b447329244d42e33b137"} Jul 02 17:06:27.609112 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:27.609055 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5jt4l" event={"ID":"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99","Type":"ContainerStarted","Data":"647ae7bb15dd32abf548ade4e3fa9d9c4ad76c4d0756e075a6b77f6a94dda41d"} Jul 02 17:06:30.625223 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:30.625132 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kmzwk" event={"ID":"f25b1961-e178-4c5e-ab99-55eaf9155a51","Type":"ContainerStarted","Data":"080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8"} Jul 02 17:06:30.626544 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:30.626520 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5jt4l" event={"ID":"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99","Type":"ContainerStarted","Data":"d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03"} Jul 02 17:06:30.641356 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:30.641311 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-kmzwk" podStartSLOduration=1.493295085 podStartE2EDuration="4.641299427s" totalImagesPullingTime="3.148004342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:06:30.639778758 +0000 UTC m=+577.829095786" watchObservedRunningTime="2026-07-02 17:06:30.641299427 +0000 UTC m=+577.830616458" Jul 02 17:06:30.656986 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:30.656655 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-5jt4l" podStartSLOduration=1.703256329 podStartE2EDuration="4.656637633s" totalImagesPullingTime="2.953381304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:06:30.656537243 +0000 UTC m=+577.845854379" watchObservedRunningTime="2026-07-02 17:06:30.656637633 +0000 UTC m=+577.845954664" Jul 02 17:06:30.684304 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:30.684272 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmzwk"] Jul 02 17:06:32.634550 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:32.634491 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-kmzwk" podUID="f25b1961-e178-4c5e-ab99-55eaf9155a51" containerName="authorino" containerID="cri-o://080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8" gracePeriod=30 Jul 02 17:06:32.877462 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:32.877440 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kmzwk" Jul 02 17:06:32.942618 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:32.942550 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtngc\" (UniqueName: \"kubernetes.io/projected/f25b1961-e178-4c5e-ab99-55eaf9155a51-kube-api-access-mtngc\") pod \"f25b1961-e178-4c5e-ab99-55eaf9155a51\" (UID: \"f25b1961-e178-4c5e-ab99-55eaf9155a51\") " Jul 02 17:06:32.944615 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:32.944591 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25b1961-e178-4c5e-ab99-55eaf9155a51-kube-api-access-mtngc" (OuterVolumeSpecName: "kube-api-access-mtngc") pod "f25b1961-e178-4c5e-ab99-55eaf9155a51" (UID: "f25b1961-e178-4c5e-ab99-55eaf9155a51"). InnerVolumeSpecName "kube-api-access-mtngc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:06:33.043092 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.043060 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtngc\" (UniqueName: \"kubernetes.io/projected/f25b1961-e178-4c5e-ab99-55eaf9155a51-kube-api-access-mtngc\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:06:33.639684 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.639649 2582 generic.go:356] "Generic (PLEG): container finished" podID="f25b1961-e178-4c5e-ab99-55eaf9155a51" containerID="080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8" exitCode=0 Jul 02 17:06:33.640119 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.639691 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kmzwk" Jul 02 17:06:33.640119 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.639703 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kmzwk" event={"ID":"f25b1961-e178-4c5e-ab99-55eaf9155a51","Type":"ContainerDied","Data":"080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8"} Jul 02 17:06:33.640119 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.639758 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kmzwk" event={"ID":"f25b1961-e178-4c5e-ab99-55eaf9155a51","Type":"ContainerDied","Data":"d92917166092973fec338e895c7678baea10d0b3d818b447329244d42e33b137"} Jul 02 17:06:33.640119 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.639780 2582 scope.go:117] "RemoveContainer" containerID="080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8" Jul 02 17:06:33.647676 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.647657 2582 scope.go:117] "RemoveContainer" containerID="080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8" Jul 02 17:06:33.647981 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:06:33.647958 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8\": container with ID starting with 080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8 not found: ID does not exist" containerID="080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8" Jul 02 17:06:33.648053 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.647990 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8"} err="failed to get container status \"080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8\": rpc error: code = NotFound desc = could not find container \"080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8\": container with ID starting with 080cd3ac60f08b9c98e2ccd11836a856874dd2896ed98b6c7b20f4b782b113e8 not found: ID does not exist" Jul 02 17:06:33.658670 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.658587 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmzwk"] Jul 02 17:06:33.660247 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:33.660227 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmzwk"] Jul 02 17:06:35.264894 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:35.264862 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f25b1961-e178-4c5e-ab99-55eaf9155a51" path="/var/lib/kubelet/pods/f25b1961-e178-4c5e-ab99-55eaf9155a51/volumes" Jul 02 17:06:53.221617 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:53.221589 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:06:53.222610 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:06:53.222591 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:07:19.572059 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.571976 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:07:19.572503 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.572262 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25b1961-e178-4c5e-ab99-55eaf9155a51" containerName="authorino" Jul 02 17:07:19.572503 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.572272 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25b1961-e178-4c5e-ab99-55eaf9155a51" containerName="authorino" Jul 02 17:07:19.572503 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.572323 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25b1961-e178-4c5e-ab99-55eaf9155a51" containerName="authorino" Jul 02 17:07:19.574451 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.574434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:19.576863 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.576844 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"default-dockercfg-bgw7c\"" Jul 02 17:07:19.577761 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.577722 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"kube-root-ca.crt\"" Jul 02 17:07:19.577862 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.577748 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jul 02 17:07:19.577998 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.577983 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"openshift-service-ca.crt\"" Jul 02 17:07:19.586838 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.586820 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:07:19.610960 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.610933 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk6d4\" (UniqueName: \"kubernetes.io/projected/1124b10b-024a-474a-8e8d-cf0e6be38126-kube-api-access-vk6d4\") pod \"maas-keycloak-0\" (UID: \"1124b10b-024a-474a-8e8d-cf0e6be38126\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:19.712061 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.712031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk6d4\" (UniqueName: \"kubernetes.io/projected/1124b10b-024a-474a-8e8d-cf0e6be38126-kube-api-access-vk6d4\") pod \"maas-keycloak-0\" (UID: \"1124b10b-024a-474a-8e8d-cf0e6be38126\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:19.720237 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.720212 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk6d4\" (UniqueName: \"kubernetes.io/projected/1124b10b-024a-474a-8e8d-cf0e6be38126-kube-api-access-vk6d4\") pod \"maas-keycloak-0\" (UID: \"1124b10b-024a-474a-8e8d-cf0e6be38126\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:19.884402 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:19.884379 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:20.013596 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:20.013489 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:07:20.016121 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:07:20.016098 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1124b10b_024a_474a_8e8d_cf0e6be38126.slice/crio-ad4350899f43a7033af5ccbb3e3d7259e88cdf323d4e7bef68d083a402e069c8 WatchSource:0}: Error finding container ad4350899f43a7033af5ccbb3e3d7259e88cdf323d4e7bef68d083a402e069c8: Status 404 returned error can't find the container with id ad4350899f43a7033af5ccbb3e3d7259e88cdf323d4e7bef68d083a402e069c8 Jul 02 17:07:20.829995 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:20.829956 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"1124b10b-024a-474a-8e8d-cf0e6be38126","Type":"ContainerStarted","Data":"ad4350899f43a7033af5ccbb3e3d7259e88cdf323d4e7bef68d083a402e069c8"} Jul 02 17:07:26.863092 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:26.863044 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"1124b10b-024a-474a-8e8d-cf0e6be38126","Type":"ContainerStarted","Data":"e4db40fb2eaa2e1186bc8a841ba33cef31fe311fb0995ed776bb1ecdd04e55a4"} Jul 02 17:07:26.885147 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:26.885104 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:26.887938 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:26.887881 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="keycloak-system/maas-keycloak-0" podStartSLOduration=2.080495738 podStartE2EDuration="7.8878598s" totalImagesPullingTime="5.807364062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:07:26.886530314 +0000 UTC m=+634.075847346" watchObservedRunningTime="2026-07-02 17:07:26.8878598 +0000 UTC m=+634.077176832" Jul 02 17:07:26.888212 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:26.888015 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:27.885235 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:27.885189 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:28.885873 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:28.885818 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:29.884802 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:29.884768 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:29.885094 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:29.885062 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:30.885161 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:30.885121 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:31.885800 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:31.885752 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:32.885677 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:32.885632 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:33.885200 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:33.885163 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:34.885014 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:34.884970 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:35.885428 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:35.885390 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:36.885185 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:36.885132 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:37.885275 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:37.885237 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.25:9000/health/started\": dial tcp 10.133.0.25:9000: connect: connection refused" Jul 02 17:07:38.982578 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:38.982543 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:38.997587 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:38.997536 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 17:07:48.988660 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:48.988630 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="keycloak-system/maas-keycloak-0" Jul 02 17:07:49.923446 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:49.923413 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6plkg"] Jul 02 17:07:49.925855 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:49.925829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6plkg" Jul 02 17:07:49.933498 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:49.933474 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6plkg"] Jul 02 17:07:50.086384 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.086345 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26svm\" (UniqueName: \"kubernetes.io/projected/ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e-kube-api-access-26svm\") pod \"authorino-5d4b67c446-6plkg\" (UID: \"ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e\") " pod="kuadrant-system/authorino-5d4b67c446-6plkg" Jul 02 17:07:50.176956 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.176873 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6plkg"] Jul 02 17:07:50.177126 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:07:50.177107 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-26svm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-6plkg" podUID="ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e" Jul 02 17:07:50.187627 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.187600 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26svm\" (UniqueName: \"kubernetes.io/projected/ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e-kube-api-access-26svm\") pod \"authorino-5d4b67c446-6plkg\" (UID: \"ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e\") " pod="kuadrant-system/authorino-5d4b67c446-6plkg" Jul 02 17:07:50.206581 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.206552 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26svm\" (UniqueName: \"kubernetes.io/projected/ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e-kube-api-access-26svm\") pod \"authorino-5d4b67c446-6plkg\" (UID: \"ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e\") " pod="kuadrant-system/authorino-5d4b67c446-6plkg" Jul 02 17:07:50.215227 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.215207 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8559f66784-wcklc"] Jul 02 17:07:50.217355 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.217340 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8559f66784-wcklc" Jul 02 17:07:50.228000 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.227975 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8559f66784-wcklc"] Jul 02 17:07:50.389406 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.389372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhqt\" (UniqueName: \"kubernetes.io/projected/c5300858-f4a1-4b8e-b1d8-8c712a79bfdf-kube-api-access-srhqt\") pod \"authorino-8559f66784-wcklc\" (UID: \"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf\") " pod="kuadrant-system/authorino-8559f66784-wcklc" Jul 02 17:07:50.490010 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.489920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhqt\" (UniqueName: \"kubernetes.io/projected/c5300858-f4a1-4b8e-b1d8-8c712a79bfdf-kube-api-access-srhqt\") pod \"authorino-8559f66784-wcklc\" (UID: \"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf\") " pod="kuadrant-system/authorino-8559f66784-wcklc" Jul 02 17:07:50.500466 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.500441 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhqt\" (UniqueName: \"kubernetes.io/projected/c5300858-f4a1-4b8e-b1d8-8c712a79bfdf-kube-api-access-srhqt\") pod \"authorino-8559f66784-wcklc\" (UID: \"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf\") " pod="kuadrant-system/authorino-8559f66784-wcklc" Jul 02 17:07:50.526335 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.526311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8559f66784-wcklc" Jul 02 17:07:50.584083 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.584025 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8559f66784-wcklc"] Jul 02 17:07:50.607571 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.607544 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9c474554d-7jl7c"] Jul 02 17:07:50.612223 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.612203 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:07:50.615068 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.615051 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 02 17:07:50.624772 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.624750 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9c474554d-7jl7c"] Jul 02 17:07:50.656929 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.656901 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8559f66784-wcklc"] Jul 02 17:07:50.659984 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:07:50.659961 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5300858_f4a1_4b8e_b1d8_8c712a79bfdf.slice/crio-b577422ea4accd0461915a54ff2aca86c3395c34a862c2e1d7a4797b49b553e2 WatchSource:0}: Error finding container b577422ea4accd0461915a54ff2aca86c3395c34a862c2e1d7a4797b49b553e2: Status 404 returned error can't find the container with id b577422ea4accd0461915a54ff2aca86c3395c34a862c2e1d7a4797b49b553e2 Jul 02 17:07:50.792518 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.792421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccade728-8136-45ba-a2a0-57d29d276976-tls-cert\") pod \"authorino-9c474554d-7jl7c\" (UID: \"ccade728-8136-45ba-a2a0-57d29d276976\") " pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:07:50.792518 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.792467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmsl\" (UniqueName: \"kubernetes.io/projected/ccade728-8136-45ba-a2a0-57d29d276976-kube-api-access-kxmsl\") pod \"authorino-9c474554d-7jl7c\" (UID: \"ccade728-8136-45ba-a2a0-57d29d276976\") " pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:07:50.893403 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.893370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccade728-8136-45ba-a2a0-57d29d276976-tls-cert\") pod \"authorino-9c474554d-7jl7c\" (UID: \"ccade728-8136-45ba-a2a0-57d29d276976\") " pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:07:50.893403 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.893408 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmsl\" (UniqueName: \"kubernetes.io/projected/ccade728-8136-45ba-a2a0-57d29d276976-kube-api-access-kxmsl\") pod \"authorino-9c474554d-7jl7c\" (UID: \"ccade728-8136-45ba-a2a0-57d29d276976\") " pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:07:50.895822 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.895800 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccade728-8136-45ba-a2a0-57d29d276976-tls-cert\") pod \"authorino-9c474554d-7jl7c\" (UID: \"ccade728-8136-45ba-a2a0-57d29d276976\") " pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:07:50.901480 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.901462 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmsl\" (UniqueName: \"kubernetes.io/projected/ccade728-8136-45ba-a2a0-57d29d276976-kube-api-access-kxmsl\") pod \"authorino-9c474554d-7jl7c\" (UID: \"ccade728-8136-45ba-a2a0-57d29d276976\") " pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:07:50.922440 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:50.922421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:07:51.000851 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:51.000524 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6plkg" Jul 02 17:07:51.000851 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:51.000539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8559f66784-wcklc" event={"ID":"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf","Type":"ContainerStarted","Data":"b577422ea4accd0461915a54ff2aca86c3395c34a862c2e1d7a4797b49b553e2"} Jul 02 17:07:51.008234 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:51.008208 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6plkg" Jul 02 17:07:51.053410 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:51.053390 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9c474554d-7jl7c"] Jul 02 17:07:51.064885 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:07:51.064855 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccade728_8136_45ba_a2a0_57d29d276976.slice/crio-aab2f7883462f469018f71ff9d9d1315f81541f7fe103ff820e3da43c5d84c2a WatchSource:0}: Error finding container aab2f7883462f469018f71ff9d9d1315f81541f7fe103ff820e3da43c5d84c2a: Status 404 returned error can't find the container with id aab2f7883462f469018f71ff9d9d1315f81541f7fe103ff820e3da43c5d84c2a Jul 02 17:07:51.095683 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:51.095663 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26svm\" (UniqueName: \"kubernetes.io/projected/ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e-kube-api-access-26svm\") pod \"ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e\" (UID: \"ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e\") " Jul 02 17:07:51.097772 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:51.097750 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e-kube-api-access-26svm" (OuterVolumeSpecName: "kube-api-access-26svm") pod "ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e" (UID: "ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e"). InnerVolumeSpecName "kube-api-access-26svm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:07:51.196827 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:51.196787 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26svm\" (UniqueName: \"kubernetes.io/projected/ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e-kube-api-access-26svm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:07:52.006915 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.006873 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8559f66784-wcklc" event={"ID":"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf","Type":"ContainerStarted","Data":"d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8"} Jul 02 17:07:52.007120 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.006919 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-8559f66784-wcklc" podUID="c5300858-f4a1-4b8e-b1d8-8c712a79bfdf" containerName="authorino" containerID="cri-o://d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8" gracePeriod=30 Jul 02 17:07:52.008617 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.008592 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6plkg" Jul 02 17:07:52.008714 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.008630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c474554d-7jl7c" event={"ID":"ccade728-8136-45ba-a2a0-57d29d276976","Type":"ContainerStarted","Data":"c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87"} Jul 02 17:07:52.008714 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.008664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c474554d-7jl7c" event={"ID":"ccade728-8136-45ba-a2a0-57d29d276976","Type":"ContainerStarted","Data":"aab2f7883462f469018f71ff9d9d1315f81541f7fe103ff820e3da43c5d84c2a"} Jul 02 17:07:52.032134 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.032096 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8559f66784-wcklc" podStartSLOduration=1.662216055 podStartE2EDuration="2.03208081s" totalImagesPullingTime="369.864755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:07:52.0309146 +0000 UTC m=+659.220231631" watchObservedRunningTime="2026-07-02 17:07:52.03208081 +0000 UTC m=+659.221397844" Jul 02 17:07:52.066632 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.066604 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6plkg"] Jul 02 17:07:52.071239 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.071213 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6plkg"] Jul 02 17:07:52.086993 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.086953 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9c474554d-7jl7c" podStartSLOduration=1.7427746480000001 podStartE2EDuration="2.086939191s" totalImagesPullingTime="344.164543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:07:52.086085356 +0000 UTC m=+659.275402388" watchObservedRunningTime="2026-07-02 17:07:52.086939191 +0000 UTC m=+659.276256222" Jul 02 17:07:52.120044 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.120009 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5jt4l"] Jul 02 17:07:52.120369 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.120253 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-5jt4l" podUID="5dbd25ed-304c-4aac-9954-b9aa9b7f2f99" containerName="authorino" containerID="cri-o://d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03" gracePeriod=30 Jul 02 17:07:52.253962 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.253939 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8559f66784-wcklc" Jul 02 17:07:52.375383 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.375356 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5jt4l" Jul 02 17:07:52.407508 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.407483 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srhqt\" (UniqueName: \"kubernetes.io/projected/c5300858-f4a1-4b8e-b1d8-8c712a79bfdf-kube-api-access-srhqt\") pod \"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf\" (UID: \"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf\") " Jul 02 17:07:52.409424 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.409395 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5300858-f4a1-4b8e-b1d8-8c712a79bfdf-kube-api-access-srhqt" (OuterVolumeSpecName: "kube-api-access-srhqt") pod "c5300858-f4a1-4b8e-b1d8-8c712a79bfdf" (UID: "c5300858-f4a1-4b8e-b1d8-8c712a79bfdf"). InnerVolumeSpecName "kube-api-access-srhqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:07:52.508432 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.508404 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-654xp\" (UniqueName: \"kubernetes.io/projected/5dbd25ed-304c-4aac-9954-b9aa9b7f2f99-kube-api-access-654xp\") pod \"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99\" (UID: \"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99\") " Jul 02 17:07:52.508601 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.508550 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhqt\" (UniqueName: \"kubernetes.io/projected/c5300858-f4a1-4b8e-b1d8-8c712a79bfdf-kube-api-access-srhqt\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:07:52.510436 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.510412 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbd25ed-304c-4aac-9954-b9aa9b7f2f99-kube-api-access-654xp" (OuterVolumeSpecName: "kube-api-access-654xp") pod "5dbd25ed-304c-4aac-9954-b9aa9b7f2f99" (UID: "5dbd25ed-304c-4aac-9954-b9aa9b7f2f99"). InnerVolumeSpecName "kube-api-access-654xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:07:52.609153 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:52.609079 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654xp\" (UniqueName: \"kubernetes.io/projected/5dbd25ed-304c-4aac-9954-b9aa9b7f2f99-kube-api-access-654xp\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:07:53.013844 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.013808 2582 generic.go:356] "Generic (PLEG): container finished" podID="5dbd25ed-304c-4aac-9954-b9aa9b7f2f99" containerID="d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03" exitCode=0 Jul 02 17:07:53.014047 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.013859 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5jt4l" Jul 02 17:07:53.014047 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.013894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5jt4l" event={"ID":"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99","Type":"ContainerDied","Data":"d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03"} Jul 02 17:07:53.014047 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.013938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5jt4l" event={"ID":"5dbd25ed-304c-4aac-9954-b9aa9b7f2f99","Type":"ContainerDied","Data":"647ae7bb15dd32abf548ade4e3fa9d9c4ad76c4d0756e075a6b77f6a94dda41d"} Jul 02 17:07:53.014047 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.013956 2582 scope.go:117] "RemoveContainer" containerID="d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03" Jul 02 17:07:53.015251 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.015136 2582 generic.go:356] "Generic (PLEG): container finished" podID="c5300858-f4a1-4b8e-b1d8-8c712a79bfdf" containerID="d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8" exitCode=0 Jul 02 17:07:53.015251 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.015183 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8559f66784-wcklc" Jul 02 17:07:53.015251 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.015213 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8559f66784-wcklc" event={"ID":"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf","Type":"ContainerDied","Data":"d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8"} Jul 02 17:07:53.015431 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.015254 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8559f66784-wcklc" event={"ID":"c5300858-f4a1-4b8e-b1d8-8c712a79bfdf","Type":"ContainerDied","Data":"b577422ea4accd0461915a54ff2aca86c3395c34a862c2e1d7a4797b49b553e2"} Jul 02 17:07:53.023293 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.023275 2582 scope.go:117] "RemoveContainer" containerID="d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03" Jul 02 17:07:53.023570 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:07:53.023551 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03\": container with ID starting with d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03 not found: ID does not exist" containerID="d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03" Jul 02 17:07:53.023608 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.023581 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03"} err="failed to get container status \"d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03\": rpc error: code = NotFound desc = could not find container \"d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03\": container with ID starting with d10af4b467869cfd675c1a47673c2baaad9d19a8fa913e7fc3ccdfd86ed1ff03 not found: ID does not exist" Jul 02 17:07:53.023608 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.023597 2582 scope.go:117] "RemoveContainer" containerID="d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8" Jul 02 17:07:53.030744 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.030712 2582 scope.go:117] "RemoveContainer" containerID="d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8" Jul 02 17:07:53.030986 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:07:53.030971 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8\": container with ID starting with d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8 not found: ID does not exist" containerID="d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8" Jul 02 17:07:53.031048 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.030995 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8"} err="failed to get container status \"d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8\": rpc error: code = NotFound desc = could not find container \"d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8\": container with ID starting with d30ae429fb14effbb96b22fcc13d20983361b9f4848498a76de2ea9b0ac1b8f8 not found: ID does not exist" Jul 02 17:07:53.034538 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.034516 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5jt4l"] Jul 02 17:07:53.040697 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.040677 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5jt4l"] Jul 02 17:07:53.050699 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.050679 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8559f66784-wcklc"] Jul 02 17:07:53.052741 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.052711 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8559f66784-wcklc"] Jul 02 17:07:53.266046 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.265991 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbd25ed-304c-4aac-9954-b9aa9b7f2f99" path="/var/lib/kubelet/pods/5dbd25ed-304c-4aac-9954-b9aa9b7f2f99/volumes" Jul 02 17:07:53.266323 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.266283 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e" path="/var/lib/kubelet/pods/ab8d57dc-73f1-4f8d-aff5-ef9ae7f4069e/volumes" Jul 02 17:07:53.266467 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:53.266455 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5300858-f4a1-4b8e-b1d8-8c712a79bfdf" path="/var/lib/kubelet/pods/c5300858-f4a1-4b8e-b1d8-8c712a79bfdf/volumes" Jul 02 17:07:56.707603 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.707567 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58b7b7c6c8-wpjrt"] Jul 02 17:07:56.707976 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.707893 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbd25ed-304c-4aac-9954-b9aa9b7f2f99" containerName="authorino" Jul 02 17:07:56.707976 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.707909 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbd25ed-304c-4aac-9954-b9aa9b7f2f99" containerName="authorino" Jul 02 17:07:56.707976 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.707920 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5300858-f4a1-4b8e-b1d8-8c712a79bfdf" containerName="authorino" Jul 02 17:07:56.707976 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.707926 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5300858-f4a1-4b8e-b1d8-8c712a79bfdf" containerName="authorino" Jul 02 17:07:56.707976 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.707966 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5300858-f4a1-4b8e-b1d8-8c712a79bfdf" containerName="authorino" Jul 02 17:07:56.707976 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.707977 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbd25ed-304c-4aac-9954-b9aa9b7f2f99" containerName="authorino" Jul 02 17:07:56.710781 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.710765 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:07:56.713693 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.713676 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8skzx\"" Jul 02 17:07:56.713955 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.713941 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 17:07:56.714038 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.713967 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 17:07:56.723581 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.723558 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58b7b7c6c8-wpjrt"] Jul 02 17:07:56.843818 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.843789 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwh9\" (UniqueName: \"kubernetes.io/projected/1f600c09-2bd8-44b6-b616-0580e6d03b1a-kube-api-access-jkwh9\") pod \"maas-controller-58b7b7c6c8-wpjrt\" (UID: \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\") " pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:07:56.843818 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.843830 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f600c09-2bd8-44b6-b616-0580e6d03b1a-webhook-cert\") pod \"maas-controller-58b7b7c6c8-wpjrt\" (UID: \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\") " pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:07:56.929182 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.929149 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qgbn7"] Jul 02 17:07:56.932244 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.932228 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:07:56.944496 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.944473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwh9\" (UniqueName: \"kubernetes.io/projected/1f600c09-2bd8-44b6-b616-0580e6d03b1a-kube-api-access-jkwh9\") pod \"maas-controller-58b7b7c6c8-wpjrt\" (UID: \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\") " pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:07:56.944597 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.944510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f600c09-2bd8-44b6-b616-0580e6d03b1a-webhook-cert\") pod \"maas-controller-58b7b7c6c8-wpjrt\" (UID: \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\") " pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:07:56.946251 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.946231 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qgbn7"] Jul 02 17:07:56.947076 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.947056 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f600c09-2bd8-44b6-b616-0580e6d03b1a-webhook-cert\") pod \"maas-controller-58b7b7c6c8-wpjrt\" (UID: \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\") " pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:07:56.961512 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:56.961456 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwh9\" (UniqueName: \"kubernetes.io/projected/1f600c09-2bd8-44b6-b616-0580e6d03b1a-kube-api-access-jkwh9\") pod \"maas-controller-58b7b7c6c8-wpjrt\" (UID: \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\") " pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:07:57.020142 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.020112 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:07:57.045264 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.045234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b753e89a-0b0e-42a4-8c73-48b620377dc0-webhook-cert\") pod \"maas-controller-7dfcd6c9c4-qgbn7\" (UID: \"b753e89a-0b0e-42a4-8c73-48b620377dc0\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:07:57.045409 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.045287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crmm\" (UniqueName: \"kubernetes.io/projected/b753e89a-0b0e-42a4-8c73-48b620377dc0-kube-api-access-6crmm\") pod \"maas-controller-7dfcd6c9c4-qgbn7\" (UID: \"b753e89a-0b0e-42a4-8c73-48b620377dc0\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:07:57.146377 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.146346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b753e89a-0b0e-42a4-8c73-48b620377dc0-webhook-cert\") pod \"maas-controller-7dfcd6c9c4-qgbn7\" (UID: \"b753e89a-0b0e-42a4-8c73-48b620377dc0\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:07:57.146530 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.146408 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crmm\" (UniqueName: \"kubernetes.io/projected/b753e89a-0b0e-42a4-8c73-48b620377dc0-kube-api-access-6crmm\") pod \"maas-controller-7dfcd6c9c4-qgbn7\" (UID: \"b753e89a-0b0e-42a4-8c73-48b620377dc0\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:07:57.148645 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.148621 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b753e89a-0b0e-42a4-8c73-48b620377dc0-webhook-cert\") pod \"maas-controller-7dfcd6c9c4-qgbn7\" (UID: \"b753e89a-0b0e-42a4-8c73-48b620377dc0\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:07:57.158621 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.158597 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58b7b7c6c8-wpjrt"] Jul 02 17:07:57.161475 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:07:57.161451 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f600c09_2bd8_44b6_b616_0580e6d03b1a.slice/crio-240c4628724d0df655d044493cb46a49a0ce71e62e2ab22ab4649fd1170c97f4 WatchSource:0}: Error finding container 240c4628724d0df655d044493cb46a49a0ce71e62e2ab22ab4649fd1170c97f4: Status 404 returned error can't find the container with id 240c4628724d0df655d044493cb46a49a0ce71e62e2ab22ab4649fd1170c97f4 Jul 02 17:07:57.163800 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.163778 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crmm\" (UniqueName: \"kubernetes.io/projected/b753e89a-0b0e-42a4-8c73-48b620377dc0-kube-api-access-6crmm\") pod \"maas-controller-7dfcd6c9c4-qgbn7\" (UID: \"b753e89a-0b0e-42a4-8c73-48b620377dc0\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:07:57.242439 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.242357 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:07:57.370583 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:57.370556 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qgbn7"] Jul 02 17:07:57.372049 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:07:57.372024 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb753e89a_0b0e_42a4_8c73_48b620377dc0.slice/crio-cbbfe5b94fff94a7d0132539fbda973470acdd2c6e5952d13ef3271300794b47 WatchSource:0}: Error finding container cbbfe5b94fff94a7d0132539fbda973470acdd2c6e5952d13ef3271300794b47: Status 404 returned error can't find the container with id cbbfe5b94fff94a7d0132539fbda973470acdd2c6e5952d13ef3271300794b47 Jul 02 17:07:58.041919 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:58.041825 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" event={"ID":"b753e89a-0b0e-42a4-8c73-48b620377dc0","Type":"ContainerStarted","Data":"cbbfe5b94fff94a7d0132539fbda973470acdd2c6e5952d13ef3271300794b47"} Jul 02 17:07:58.043692 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:07:58.043665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" event={"ID":"1f600c09-2bd8-44b6-b616-0580e6d03b1a","Type":"ContainerStarted","Data":"240c4628724d0df655d044493cb46a49a0ce71e62e2ab22ab4649fd1170c97f4"} Jul 02 17:08:00.055108 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:00.055076 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" event={"ID":"b753e89a-0b0e-42a4-8c73-48b620377dc0","Type":"ContainerStarted","Data":"eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1"} Jul 02 17:08:00.055539 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:00.055199 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:08:00.056556 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:00.056529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" event={"ID":"1f600c09-2bd8-44b6-b616-0580e6d03b1a","Type":"ContainerStarted","Data":"4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e"} Jul 02 17:08:00.056688 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:00.056614 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:08:00.075803 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:00.075764 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" podStartSLOduration=2.244716904 podStartE2EDuration="4.075748502s" totalImagesPullingTime="1.831031598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:08:00.074477358 +0000 UTC m=+667.263794390" watchObservedRunningTime="2026-07-02 17:08:00.075748502 +0000 UTC m=+667.265065534" Jul 02 17:08:00.093270 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:00.093232 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" podStartSLOduration=2.054094692 podStartE2EDuration="4.093221881s" totalImagesPullingTime="2.039127189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:08:00.092394782 +0000 UTC m=+667.281711814" watchObservedRunningTime="2026-07-02 17:08:00.093221881 +0000 UTC m=+667.282538912" Jul 02 17:08:11.066993 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.066960 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:08:11.067404 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.067052 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:08:11.131901 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.131867 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58b7b7c6c8-wpjrt"] Jul 02 17:08:11.132075 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.132039 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" podUID="1f600c09-2bd8-44b6-b616-0580e6d03b1a" containerName="manager" containerID="cri-o://4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e" gracePeriod=30 Jul 02 17:08:11.380635 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.380613 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:08:11.462341 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.462311 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f600c09-2bd8-44b6-b616-0580e6d03b1a-webhook-cert\") pod \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\" (UID: \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\") " Jul 02 17:08:11.462507 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.462372 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkwh9\" (UniqueName: \"kubernetes.io/projected/1f600c09-2bd8-44b6-b616-0580e6d03b1a-kube-api-access-jkwh9\") pod \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\" (UID: \"1f600c09-2bd8-44b6-b616-0580e6d03b1a\") " Jul 02 17:08:11.464514 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.464478 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f600c09-2bd8-44b6-b616-0580e6d03b1a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1f600c09-2bd8-44b6-b616-0580e6d03b1a" (UID: "1f600c09-2bd8-44b6-b616-0580e6d03b1a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:08:11.464661 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.464596 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f600c09-2bd8-44b6-b616-0580e6d03b1a-kube-api-access-jkwh9" (OuterVolumeSpecName: "kube-api-access-jkwh9") pod "1f600c09-2bd8-44b6-b616-0580e6d03b1a" (UID: "1f600c09-2bd8-44b6-b616-0580e6d03b1a"). InnerVolumeSpecName "kube-api-access-jkwh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:08:11.562962 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.562930 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f600c09-2bd8-44b6-b616-0580e6d03b1a-webhook-cert\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:08:11.562962 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:11.562959 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkwh9\" (UniqueName: \"kubernetes.io/projected/1f600c09-2bd8-44b6-b616-0580e6d03b1a-kube-api-access-jkwh9\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:08:12.111444 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.111409 2582 generic.go:356] "Generic (PLEG): container finished" podID="1f600c09-2bd8-44b6-b616-0580e6d03b1a" containerID="4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e" exitCode=0 Jul 02 17:08:12.111915 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.111462 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" Jul 02 17:08:12.111915 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.111494 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" event={"ID":"1f600c09-2bd8-44b6-b616-0580e6d03b1a","Type":"ContainerDied","Data":"4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e"} Jul 02 17:08:12.111915 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.111570 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b7b7c6c8-wpjrt" event={"ID":"1f600c09-2bd8-44b6-b616-0580e6d03b1a","Type":"ContainerDied","Data":"240c4628724d0df655d044493cb46a49a0ce71e62e2ab22ab4649fd1170c97f4"} Jul 02 17:08:12.111915 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.111592 2582 scope.go:117] "RemoveContainer" containerID="4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e" Jul 02 17:08:12.120070 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.120054 2582 scope.go:117] "RemoveContainer" containerID="4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e" Jul 02 17:08:12.120302 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:08:12.120285 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e\": container with ID starting with 4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e not found: ID does not exist" containerID="4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e" Jul 02 17:08:12.120345 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.120310 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e"} err="failed to get container status \"4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e\": rpc error: code = NotFound desc = could not find container \"4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e\": container with ID starting with 4f6194b3f13d3c340104c35ed88a7aeae66e1ec188e12854d83a776965df189e not found: ID does not exist" Jul 02 17:08:12.132267 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.132240 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58b7b7c6c8-wpjrt"] Jul 02 17:08:12.137239 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:12.137219 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58b7b7c6c8-wpjrt"] Jul 02 17:08:13.265058 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:13.265026 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f600c09-2bd8-44b6-b616-0580e6d03b1a" path="/var/lib/kubelet/pods/1f600c09-2bd8-44b6-b616-0580e6d03b1a/volumes" Jul 02 17:08:33.096869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.096834 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g"] Jul 02 17:08:33.097328 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.097156 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f600c09-2bd8-44b6-b616-0580e6d03b1a" containerName="manager" Jul 02 17:08:33.097328 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.097168 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f600c09-2bd8-44b6-b616-0580e6d03b1a" containerName="manager" Jul 02 17:08:33.097328 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.097212 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f600c09-2bd8-44b6-b616-0580e6d03b1a" containerName="manager" Jul 02 17:08:33.102679 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.102662 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:33.105677 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.105649 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 02 17:08:33.105985 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.105959 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rb8w2\"" Jul 02 17:08:33.112250 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.112223 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g"] Jul 02 17:08:33.203628 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.203595 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bbcb654f4-w468k"] Jul 02 17:08:33.207736 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.207705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:33.217506 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.217485 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bbcb654f4-w468k"] Jul 02 17:08:33.239968 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.239941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdpbn\" (UniqueName: \"kubernetes.io/projected/804c7c00-4f66-49a7-8219-bece72c63475-kube-api-access-tdpbn\") pod \"payload-pre-processing-59f49fd4f6-25l5g\" (UID: \"804c7c00-4f66-49a7-8219-bece72c63475\") " pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:33.240078 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.239992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804c7c00-4f66-49a7-8219-bece72c63475-plugins-config-volume\") pod \"payload-pre-processing-59f49fd4f6-25l5g\" (UID: \"804c7c00-4f66-49a7-8219-bece72c63475\") " pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:33.341135 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.341100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdpbn\" (UniqueName: \"kubernetes.io/projected/804c7c00-4f66-49a7-8219-bece72c63475-kube-api-access-tdpbn\") pod \"payload-pre-processing-59f49fd4f6-25l5g\" (UID: \"804c7c00-4f66-49a7-8219-bece72c63475\") " pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:33.341297 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.341144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041b935c-fe2b-460e-a549-cc0189cf0963-plugins-config-volume\") pod \"payload-processing-7bbcb654f4-w468k\" (UID: \"041b935c-fe2b-460e-a549-cc0189cf0963\") " pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:33.341297 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.341162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54q7\" (UniqueName: \"kubernetes.io/projected/041b935c-fe2b-460e-a549-cc0189cf0963-kube-api-access-j54q7\") pod \"payload-processing-7bbcb654f4-w468k\" (UID: \"041b935c-fe2b-460e-a549-cc0189cf0963\") " pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:33.341297 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.341255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804c7c00-4f66-49a7-8219-bece72c63475-plugins-config-volume\") pod \"payload-pre-processing-59f49fd4f6-25l5g\" (UID: \"804c7c00-4f66-49a7-8219-bece72c63475\") " pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:33.341961 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.341944 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804c7c00-4f66-49a7-8219-bece72c63475-plugins-config-volume\") pod \"payload-pre-processing-59f49fd4f6-25l5g\" (UID: \"804c7c00-4f66-49a7-8219-bece72c63475\") " pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:33.349444 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.349386 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdpbn\" (UniqueName: \"kubernetes.io/projected/804c7c00-4f66-49a7-8219-bece72c63475-kube-api-access-tdpbn\") pod \"payload-pre-processing-59f49fd4f6-25l5g\" (UID: \"804c7c00-4f66-49a7-8219-bece72c63475\") " pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:33.414104 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.413322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:33.442581 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.442550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041b935c-fe2b-460e-a549-cc0189cf0963-plugins-config-volume\") pod \"payload-processing-7bbcb654f4-w468k\" (UID: \"041b935c-fe2b-460e-a549-cc0189cf0963\") " pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:33.442757 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.442593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54q7\" (UniqueName: \"kubernetes.io/projected/041b935c-fe2b-460e-a549-cc0189cf0963-kube-api-access-j54q7\") pod \"payload-processing-7bbcb654f4-w468k\" (UID: \"041b935c-fe2b-460e-a549-cc0189cf0963\") " pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:33.443243 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.443223 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041b935c-fe2b-460e-a549-cc0189cf0963-plugins-config-volume\") pod \"payload-processing-7bbcb654f4-w468k\" (UID: \"041b935c-fe2b-460e-a549-cc0189cf0963\") " pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:33.450746 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.450702 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54q7\" (UniqueName: \"kubernetes.io/projected/041b935c-fe2b-460e-a549-cc0189cf0963-kube-api-access-j54q7\") pod \"payload-processing-7bbcb654f4-w468k\" (UID: \"041b935c-fe2b-460e-a549-cc0189cf0963\") " pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:33.518156 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.518094 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:33.546714 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.546689 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g"] Jul 02 17:08:33.549452 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:08:33.549422 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804c7c00_4f66_49a7_8219_bece72c63475.slice/crio-d8d8322b4df4046c464b5503ed7bdcf84d01889f871510b03221c44321db351f WatchSource:0}: Error finding container d8d8322b4df4046c464b5503ed7bdcf84d01889f871510b03221c44321db351f: Status 404 returned error can't find the container with id d8d8322b4df4046c464b5503ed7bdcf84d01889f871510b03221c44321db351f Jul 02 17:08:33.551555 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.551533 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 17:08:33.653247 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:33.653221 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bbcb654f4-w468k"] Jul 02 17:08:33.654649 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:08:33.654622 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041b935c_fe2b_460e_a549_cc0189cf0963.slice/crio-c25f214f9315ea4d2e17392ca7f0b2069097f7e5e3ac009df339e1901d755eb3 WatchSource:0}: Error finding container c25f214f9315ea4d2e17392ca7f0b2069097f7e5e3ac009df339e1901d755eb3: Status 404 returned error can't find the container with id c25f214f9315ea4d2e17392ca7f0b2069097f7e5e3ac009df339e1901d755eb3 Jul 02 17:08:34.215305 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:34.215269 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" event={"ID":"804c7c00-4f66-49a7-8219-bece72c63475","Type":"ContainerStarted","Data":"d8d8322b4df4046c464b5503ed7bdcf84d01889f871510b03221c44321db351f"} Jul 02 17:08:34.218128 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:34.218097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" event={"ID":"041b935c-fe2b-460e-a549-cc0189cf0963","Type":"ContainerStarted","Data":"c25f214f9315ea4d2e17392ca7f0b2069097f7e5e3ac009df339e1901d755eb3"} Jul 02 17:08:36.230352 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:36.230294 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" event={"ID":"041b935c-fe2b-460e-a549-cc0189cf0963","Type":"ContainerStarted","Data":"99dc2233ffc5c357219577e889d366325ded4e6ac48775f5e77ede946dfff2d3"} Jul 02 17:08:36.230807 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:36.230407 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:36.233122 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:36.233094 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" event={"ID":"804c7c00-4f66-49a7-8219-bece72c63475","Type":"ContainerStarted","Data":"63a65b4cb2bfb74aedded02fbd2b4ea5d277bd9f890e8e3091df17fc03603e5c"} Jul 02 17:08:36.233537 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:36.233449 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:36.246164 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:36.246074 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" podStartSLOduration=0.910416603 podStartE2EDuration="3.246061285s" totalImagesPullingTime="2.335644682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:08:36.245200437 +0000 UTC m=+703.434517466" watchObservedRunningTime="2026-07-02 17:08:36.246061285 +0000 UTC m=+703.435378294" Jul 02 17:08:36.262692 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:36.262645 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" podStartSLOduration=0.814960096 podStartE2EDuration="3.262630529s" totalImagesPullingTime="2.447670433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:08:36.26084972 +0000 UTC m=+703.450166752" watchObservedRunningTime="2026-07-02 17:08:36.262630529 +0000 UTC m=+703.451947561" Jul 02 17:08:43.250463 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:43.250428 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:08:43.250979 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:43.250646 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="keycloak-system/maas-keycloak-0" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" containerID="cri-o://e4db40fb2eaa2e1186bc8a841ba33cef31fe311fb0995ed776bb1ecdd04e55a4" gracePeriod=30 Jul 02 17:08:45.276791 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:45.276766 2582 generic.go:356] "Generic (PLEG): container finished" podID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerID="e4db40fb2eaa2e1186bc8a841ba33cef31fe311fb0995ed776bb1ecdd04e55a4" exitCode=143 Jul 02 17:08:45.277110 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:45.276839 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"1124b10b-024a-474a-8e8d-cf0e6be38126","Type":"ContainerDied","Data":"e4db40fb2eaa2e1186bc8a841ba33cef31fe311fb0995ed776bb1ecdd04e55a4"} Jul 02 17:08:45.277110 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:45.276884 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"1124b10b-024a-474a-8e8d-cf0e6be38126","Type":"ContainerDied","Data":"ad4350899f43a7033af5ccbb3e3d7259e88cdf323d4e7bef68d083a402e069c8"} Jul 02 17:08:45.277110 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:45.276898 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4350899f43a7033af5ccbb3e3d7259e88cdf323d4e7bef68d083a402e069c8" Jul 02 17:08:45.289187 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:45.289168 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:45.446841 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:45.446751 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk6d4\" (UniqueName: \"kubernetes.io/projected/1124b10b-024a-474a-8e8d-cf0e6be38126-kube-api-access-vk6d4\") pod \"1124b10b-024a-474a-8e8d-cf0e6be38126\" (UID: \"1124b10b-024a-474a-8e8d-cf0e6be38126\") " Jul 02 17:08:45.448919 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:45.448896 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1124b10b-024a-474a-8e8d-cf0e6be38126-kube-api-access-vk6d4" (OuterVolumeSpecName: "kube-api-access-vk6d4") pod "1124b10b-024a-474a-8e8d-cf0e6be38126" (UID: "1124b10b-024a-474a-8e8d-cf0e6be38126"). InnerVolumeSpecName "kube-api-access-vk6d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:08:45.547974 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:45.547941 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk6d4\" (UniqueName: \"kubernetes.io/projected/1124b10b-024a-474a-8e8d-cf0e6be38126-kube-api-access-vk6d4\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:08:46.280599 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.280560 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.300328 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.300302 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:08:46.304565 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.304541 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:08:46.327909 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.327886 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:08:46.328204 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.328192 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" Jul 02 17:08:46.328245 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.328207 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" Jul 02 17:08:46.328278 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.328268 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" containerName="keycloak" Jul 02 17:08:46.331100 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.331082 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.333771 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.333754 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"openshift-service-ca.crt\"" Jul 02 17:08:46.333860 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.333768 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"kube-root-ca.crt\"" Jul 02 17:08:46.333860 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.333816 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"default-dockercfg-bgw7c\"" Jul 02 17:08:46.333860 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.333828 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jul 02 17:08:46.333996 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.333879 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"keycloak-test-realms\"" Jul 02 17:08:46.344384 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.344365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:08:46.454773 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.454713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25lw\" (UniqueName: \"kubernetes.io/projected/7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f-kube-api-access-k25lw\") pod \"maas-keycloak-0\" (UID: \"7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.454941 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.454783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f-test-realms\") pod \"maas-keycloak-0\" (UID: \"7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.556191 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.556107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25lw\" (UniqueName: \"kubernetes.io/projected/7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f-kube-api-access-k25lw\") pod \"maas-keycloak-0\" (UID: \"7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.556191 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.556151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f-test-realms\") pod \"maas-keycloak-0\" (UID: \"7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.556906 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.556881 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f-test-realms\") pod \"maas-keycloak-0\" (UID: \"7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.564863 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.564839 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25lw\" (UniqueName: \"kubernetes.io/projected/7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f-kube-api-access-k25lw\") pod \"maas-keycloak-0\" (UID: \"7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f\") " pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.641048 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.641019 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:46.769584 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:46.769556 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 02 17:08:46.772846 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:08:46.772823 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8a4d7d_b6e6_4e1f_9c9c_481a866ae60f.slice/crio-e48b476fa2f3061dd5bd9da650543411cd95443e24eb993b70ae4efc1d267668 WatchSource:0}: Error finding container e48b476fa2f3061dd5bd9da650543411cd95443e24eb993b70ae4efc1d267668: Status 404 returned error can't find the container with id e48b476fa2f3061dd5bd9da650543411cd95443e24eb993b70ae4efc1d267668 Jul 02 17:08:47.240907 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:47.240878 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bbcb654f4-w468k" Jul 02 17:08:47.241055 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:47.240962 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59f49fd4f6-25l5g" Jul 02 17:08:47.266231 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:47.266193 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1124b10b-024a-474a-8e8d-cf0e6be38126" path="/var/lib/kubelet/pods/1124b10b-024a-474a-8e8d-cf0e6be38126/volumes" Jul 02 17:08:47.287808 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:47.287774 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f","Type":"ContainerStarted","Data":"80acf3a81dfe4e84a642d08933b1b9196e458192dacc5c0728db32bcb294bddd"} Jul 02 17:08:47.288165 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:47.287819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f","Type":"ContainerStarted","Data":"e48b476fa2f3061dd5bd9da650543411cd95443e24eb993b70ae4efc1d267668"} Jul 02 17:08:47.332894 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:47.332846 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="keycloak-system/maas-keycloak-0" podStartSLOduration=0.998662029 podStartE2EDuration="1.332829057s" totalImagesPullingTime="334.167028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:08:47.331155288 +0000 UTC m=+714.520472321" watchObservedRunningTime="2026-07-02 17:08:47.332829057 +0000 UTC m=+714.522146088" Jul 02 17:08:47.642179 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:47.642132 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:47.643369 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:47.643331 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:48.642373 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:48.642323 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:49.642032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:49.641982 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:50.642197 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:50.642139 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:51.642037 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:51.641978 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:52.642266 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:52.642208 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:53.642024 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:53.641964 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:54.642136 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:54.642079 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:55.642268 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:55.642223 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:56.641616 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:56.641572 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="keycloak-system/maas-keycloak-0" Jul 02 17:08:56.641919 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:56.641884 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:57.641746 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:57.641687 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:58.641813 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:58.641775 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:08:59.641414 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:08:59.641372 2582 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.33:9000/health/started\": dial tcp 10.133.0.33:9000: connect: connection refused" Jul 02 17:09:00.738744 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:00.738677 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="keycloak-system/maas-keycloak-0" Jul 02 17:09:00.752914 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:00.752871 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="keycloak-system/maas-keycloak-0" podUID="7d8a4d7d-b6e6-4e1f-9c9c-481a866ae60f" containerName="keycloak" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 17:09:10.745625 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:10.745588 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="keycloak-system/maas-keycloak-0" Jul 02 17:09:21.481435 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.481405 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cd78889d6-cjph5"] Jul 02 17:09:21.484744 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.484707 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.487097 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.487081 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"authorino-oidc-ca\"" Jul 02 17:09:21.497375 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.497338 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cd78889d6-cjph5"] Jul 02 17:09:21.602749 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.602690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119393ff-51df-4c90-a0cc-0a68b99ecad9-tls-cert\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.602749 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.602752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/119393ff-51df-4c90-a0cc-0a68b99ecad9-oidc-ca\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.602938 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.602792 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwfkl\" (UniqueName: \"kubernetes.io/projected/119393ff-51df-4c90-a0cc-0a68b99ecad9-kube-api-access-rwfkl\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.704051 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.704017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119393ff-51df-4c90-a0cc-0a68b99ecad9-tls-cert\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.704051 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.704050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/119393ff-51df-4c90-a0cc-0a68b99ecad9-oidc-ca\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.704274 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.704070 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwfkl\" (UniqueName: \"kubernetes.io/projected/119393ff-51df-4c90-a0cc-0a68b99ecad9-kube-api-access-rwfkl\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.704666 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.704644 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/119393ff-51df-4c90-a0cc-0a68b99ecad9-oidc-ca\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.706621 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.706597 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119393ff-51df-4c90-a0cc-0a68b99ecad9-tls-cert\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.712361 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.712335 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwfkl\" (UniqueName: \"kubernetes.io/projected/119393ff-51df-4c90-a0cc-0a68b99ecad9-kube-api-access-rwfkl\") pod \"authorino-7cd78889d6-cjph5\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.793247 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.793171 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:09:21.932900 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:21.932873 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cd78889d6-cjph5"] Jul 02 17:09:21.935679 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:09:21.935652 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod119393ff_51df_4c90_a0cc_0a68b99ecad9.slice/crio-2e954b82a967ba8d4b7acba6e618f5b3153c5492364103e9558a1e74dde9323f WatchSource:0}: Error finding container 2e954b82a967ba8d4b7acba6e618f5b3153c5492364103e9558a1e74dde9323f: Status 404 returned error can't find the container with id 2e954b82a967ba8d4b7acba6e618f5b3153c5492364103e9558a1e74dde9323f Jul 02 17:09:22.474751 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.474698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cd78889d6-cjph5" event={"ID":"119393ff-51df-4c90-a0cc-0a68b99ecad9","Type":"ContainerStarted","Data":"40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1"} Jul 02 17:09:22.474919 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.474766 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cd78889d6-cjph5" event={"ID":"119393ff-51df-4c90-a0cc-0a68b99ecad9","Type":"ContainerStarted","Data":"2e954b82a967ba8d4b7acba6e618f5b3153c5492364103e9558a1e74dde9323f"} Jul 02 17:09:22.492258 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.492205 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cd78889d6-cjph5" podStartSLOduration=1.090744479 podStartE2EDuration="1.492189591s" totalImagesPullingTime="401.445112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:09:22.490317577 +0000 UTC m=+749.679634608" watchObservedRunningTime="2026-07-02 17:09:22.492189591 +0000 UTC m=+749.681506622" Jul 02 17:09:22.515908 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.515880 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9c474554d-7jl7c"] Jul 02 17:09:22.516067 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.516034 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-9c474554d-7jl7c" podUID="ccade728-8136-45ba-a2a0-57d29d276976" containerName="authorino" containerID="cri-o://c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87" gracePeriod=30 Jul 02 17:09:22.756856 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.756835 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:09:22.915862 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.915828 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccade728-8136-45ba-a2a0-57d29d276976-tls-cert\") pod \"ccade728-8136-45ba-a2a0-57d29d276976\" (UID: \"ccade728-8136-45ba-a2a0-57d29d276976\") " Jul 02 17:09:22.916057 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.915876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxmsl\" (UniqueName: \"kubernetes.io/projected/ccade728-8136-45ba-a2a0-57d29d276976-kube-api-access-kxmsl\") pod \"ccade728-8136-45ba-a2a0-57d29d276976\" (UID: \"ccade728-8136-45ba-a2a0-57d29d276976\") " Jul 02 17:09:22.917937 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.917906 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccade728-8136-45ba-a2a0-57d29d276976-kube-api-access-kxmsl" (OuterVolumeSpecName: "kube-api-access-kxmsl") pod "ccade728-8136-45ba-a2a0-57d29d276976" (UID: "ccade728-8136-45ba-a2a0-57d29d276976"). InnerVolumeSpecName "kube-api-access-kxmsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:09:22.925451 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:22.925428 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccade728-8136-45ba-a2a0-57d29d276976-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ccade728-8136-45ba-a2a0-57d29d276976" (UID: "ccade728-8136-45ba-a2a0-57d29d276976"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:09:23.016955 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.016917 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ccade728-8136-45ba-a2a0-57d29d276976-tls-cert\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:09:23.016955 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.016947 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmsl\" (UniqueName: \"kubernetes.io/projected/ccade728-8136-45ba-a2a0-57d29d276976-kube-api-access-kxmsl\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:09:23.479404 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.479369 2582 generic.go:356] "Generic (PLEG): container finished" podID="ccade728-8136-45ba-a2a0-57d29d276976" containerID="c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87" exitCode=0 Jul 02 17:09:23.479597 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.479415 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c474554d-7jl7c" Jul 02 17:09:23.479597 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.479431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c474554d-7jl7c" event={"ID":"ccade728-8136-45ba-a2a0-57d29d276976","Type":"ContainerDied","Data":"c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87"} Jul 02 17:09:23.479597 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.479476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c474554d-7jl7c" event={"ID":"ccade728-8136-45ba-a2a0-57d29d276976","Type":"ContainerDied","Data":"aab2f7883462f469018f71ff9d9d1315f81541f7fe103ff820e3da43c5d84c2a"} Jul 02 17:09:23.479597 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.479494 2582 scope.go:117] "RemoveContainer" containerID="c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87" Jul 02 17:09:23.487670 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.487652 2582 scope.go:117] "RemoveContainer" containerID="c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87" Jul 02 17:09:23.487940 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:09:23.487922 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87\": container with ID starting with c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87 not found: ID does not exist" containerID="c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87" Jul 02 17:09:23.487986 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.487949 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87"} err="failed to get container status \"c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87\": rpc error: code = NotFound desc = could not find container \"c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87\": container with ID starting with c44dc54661bb4e82f86245e036228f9657fd4d0bae5e02d235b3889dc8ca4a87 not found: ID does not exist" Jul 02 17:09:23.498137 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.498111 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9c474554d-7jl7c"] Jul 02 17:09:23.502482 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:23.502459 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-9c474554d-7jl7c"] Jul 02 17:09:25.266345 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:25.266310 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccade728-8136-45ba-a2a0-57d29d276976" path="/var/lib/kubelet/pods/ccade728-8136-45ba-a2a0-57d29d276976/volumes" Jul 02 17:09:52.964707 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.964649 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj"] Jul 02 17:09:52.965318 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.964997 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccade728-8136-45ba-a2a0-57d29d276976" containerName="authorino" Jul 02 17:09:52.965318 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.965013 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccade728-8136-45ba-a2a0-57d29d276976" containerName="authorino" Jul 02 17:09:52.965318 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.965085 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccade728-8136-45ba-a2a0-57d29d276976" containerName="authorino" Jul 02 17:09:52.967814 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.967794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:52.971913 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.971891 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 17:09:52.972098 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.971916 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 17:09:52.972214 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.971935 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 02 17:09:52.972214 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.971987 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n9z5j\"" Jul 02 17:09:52.980397 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:52.980367 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj"] Jul 02 17:09:53.044673 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.044632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.044673 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.044677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffrd\" (UniqueName: \"kubernetes.io/projected/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-kube-api-access-6ffrd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.044946 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.044805 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.044946 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.044890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.044946 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.044921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.044946 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.044938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.145973 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.145938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.146138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.145984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.146138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.146001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.146138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.146020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.146138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.146089 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.146138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.146114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffrd\" (UniqueName: \"kubernetes.io/projected/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-kube-api-access-6ffrd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.146390 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.146366 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.146439 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.146397 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.146523 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.146506 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.148385 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.148361 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.148555 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.148536 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.154622 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.154601 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffrd\" (UniqueName: \"kubernetes.io/projected/14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9-kube-api-access-6ffrd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nw6vj\" (UID: \"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.280928 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.280906 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n9z5j\"" Jul 02 17:09:53.289216 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.289188 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:09:53.417678 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.417618 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj"] Jul 02 17:09:53.419811 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:09:53.419783 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e8de8b_6ca2_4445_8977_cc4dd5cc7ba9.slice/crio-af56d0cd22f8ac9e7e8b82254308320ac639f6654715ead9a19c54c08a5071b6 WatchSource:0}: Error finding container af56d0cd22f8ac9e7e8b82254308320ac639f6654715ead9a19c54c08a5071b6: Status 404 returned error can't find the container with id af56d0cd22f8ac9e7e8b82254308320ac639f6654715ead9a19c54c08a5071b6 Jul 02 17:09:53.613073 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:53.612987 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" event={"ID":"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9","Type":"ContainerStarted","Data":"af56d0cd22f8ac9e7e8b82254308320ac639f6654715ead9a19c54c08a5071b6"} Jul 02 17:09:55.624812 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:55.624777 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" event={"ID":"14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9","Type":"ContainerStarted","Data":"ad8b3469bb20ba460727de66a32cfbeee27c00797146b46a97c49d137250b736"} Jul 02 17:09:55.643289 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:55.643237 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" podStartSLOduration=2.384966934 podStartE2EDuration="3.643219895s" totalImagesPullingTime="1.258252961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:09:55.642006634 +0000 UTC m=+782.831323667" watchObservedRunningTime="2026-07-02 17:09:55.643219895 +0000 UTC m=+782.832536971" Jul 02 17:09:59.875585 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:59.875549 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn"] Jul 02 17:09:59.879894 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:59.879870 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:09:59.882796 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:59.882776 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 02 17:09:59.890430 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:09:59.890406 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn"] Jul 02 17:10:00.004577 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.004542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abdc36a7-6bf4-4486-83c3-328c070a9f08-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.004577 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.004583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.004831 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.004617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.004831 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.004661 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.004831 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.004677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjsq\" (UniqueName: \"kubernetes.io/projected/abdc36a7-6bf4-4486-83c3-328c070a9f08-kube-api-access-mcjsq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.004831 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.004768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.105479 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.105441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.105631 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.105501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abdc36a7-6bf4-4486-83c3-328c070a9f08-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.105631 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.105523 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.105631 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.105559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.105631 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.105575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.105631 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.105589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjsq\" (UniqueName: \"kubernetes.io/projected/abdc36a7-6bf4-4486-83c3-328c070a9f08-kube-api-access-mcjsq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.105994 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.105957 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.105994 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.105984 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.106150 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.106011 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.107877 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.107851 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abdc36a7-6bf4-4486-83c3-328c070a9f08-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.108077 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.108058 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abdc36a7-6bf4-4486-83c3-328c070a9f08-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.115032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.115008 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjsq\" (UniqueName: \"kubernetes.io/projected/abdc36a7-6bf4-4486-83c3-328c070a9f08-kube-api-access-mcjsq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vxktn\" (UID: \"abdc36a7-6bf4-4486-83c3-328c070a9f08\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.191091 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.191056 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:00.322505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.322479 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn"] Jul 02 17:10:00.324703 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:10:00.324672 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdc36a7_6bf4_4486_83c3_328c070a9f08.slice/crio-394ea790f99c150cf546f88708a8bae1707cf12b03f2d17bc4d27df08d49b954 WatchSource:0}: Error finding container 394ea790f99c150cf546f88708a8bae1707cf12b03f2d17bc4d27df08d49b954: Status 404 returned error can't find the container with id 394ea790f99c150cf546f88708a8bae1707cf12b03f2d17bc4d27df08d49b954 Jul 02 17:10:00.646608 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:00.646583 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" event={"ID":"abdc36a7-6bf4-4486-83c3-328c070a9f08","Type":"ContainerStarted","Data":"394ea790f99c150cf546f88708a8bae1707cf12b03f2d17bc4d27df08d49b954"} Jul 02 17:10:01.652666 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:01.652630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" event={"ID":"abdc36a7-6bf4-4486-83c3-328c070a9f08","Type":"ContainerStarted","Data":"4a5464f4ce7f6fc34ece53f69bd669646b537c4aafda6230c93e9abebd18c458"} Jul 02 17:10:01.669937 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:01.669884 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" podStartSLOduration=2.373940552 podStartE2EDuration="2.669865003s" totalImagesPullingTime="295.924451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:10:01.668204499 +0000 UTC m=+788.857521530" watchObservedRunningTime="2026-07-02 17:10:01.669865003 +0000 UTC m=+788.859182013" Jul 02 17:10:03.289754 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:03.289688 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:10:03.290154 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:03.289781 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:10:03.302197 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:03.302167 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:10:03.672879 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:03.672845 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nw6vj" Jul 02 17:10:10.191852 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:10.191815 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:10.192240 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:10.191872 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:10.204250 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:10.204226 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:10:10.703198 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:10:10.703164 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vxktn" Jul 02 17:11:41.048647 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.048610 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bb8b8694c-2sqvd"] Jul 02 17:11:41.051772 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.051751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.060262 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.060234 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bb8b8694c-2sqvd"] Jul 02 17:11:41.141057 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.141029 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/3ab7866b-0b49-4646-9e07-041ea86b56d8-oidc-ca\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.141179 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.141062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ab7866b-0b49-4646-9e07-041ea86b56d8-tls-cert\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.141179 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.141094 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlqh\" (UniqueName: \"kubernetes.io/projected/3ab7866b-0b49-4646-9e07-041ea86b56d8-kube-api-access-vdlqh\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.242123 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.242091 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/3ab7866b-0b49-4646-9e07-041ea86b56d8-oidc-ca\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.242123 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.242127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ab7866b-0b49-4646-9e07-041ea86b56d8-tls-cert\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.242326 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.242265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlqh\" (UniqueName: \"kubernetes.io/projected/3ab7866b-0b49-4646-9e07-041ea86b56d8-kube-api-access-vdlqh\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.242868 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.242850 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/3ab7866b-0b49-4646-9e07-041ea86b56d8-oidc-ca\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.244494 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.244475 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ab7866b-0b49-4646-9e07-041ea86b56d8-tls-cert\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.250130 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.250106 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlqh\" (UniqueName: \"kubernetes.io/projected/3ab7866b-0b49-4646-9e07-041ea86b56d8-kube-api-access-vdlqh\") pod \"authorino-5bb8b8694c-2sqvd\" (UID: \"3ab7866b-0b49-4646-9e07-041ea86b56d8\") " pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.380662 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.380584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" Jul 02 17:11:41.519524 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:41.519498 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bb8b8694c-2sqvd"] Jul 02 17:11:41.522036 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:11:41.522006 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ab7866b_0b49_4646_9e07_041ea86b56d8.slice/crio-df50c1b821f4faeb823300569b632e5bb579509bf24c0de887fccb739e9091b4 WatchSource:0}: Error finding container df50c1b821f4faeb823300569b632e5bb579509bf24c0de887fccb739e9091b4: Status 404 returned error can't find the container with id df50c1b821f4faeb823300569b632e5bb579509bf24c0de887fccb739e9091b4 Jul 02 17:11:42.093272 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.093235 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" event={"ID":"3ab7866b-0b49-4646-9e07-041ea86b56d8","Type":"ContainerStarted","Data":"c41111b3804b05ed7cbb4a6300b8408d99e51304b6005e231d131720ca89eaab"} Jul 02 17:11:42.093644 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.093280 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" event={"ID":"3ab7866b-0b49-4646-9e07-041ea86b56d8","Type":"ContainerStarted","Data":"df50c1b821f4faeb823300569b632e5bb579509bf24c0de887fccb739e9091b4"} Jul 02 17:11:42.110193 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.110132 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5bb8b8694c-2sqvd" podStartSLOduration=0.739787411 podStartE2EDuration="1.110116153s" totalImagesPullingTime="370.328742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:11:42.108841068 +0000 UTC m=+889.298158100" watchObservedRunningTime="2026-07-02 17:11:42.110116153 +0000 UTC m=+889.299433247" Jul 02 17:11:42.141431 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.141369 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cd78889d6-cjph5"] Jul 02 17:11:42.141744 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.141586 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7cd78889d6-cjph5" podUID="119393ff-51df-4c90-a0cc-0a68b99ecad9" containerName="authorino" containerID="cri-o://40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1" gracePeriod=30 Jul 02 17:11:42.407783 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.407756 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:11:42.453473 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.453379 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119393ff-51df-4c90-a0cc-0a68b99ecad9-tls-cert\") pod \"119393ff-51df-4c90-a0cc-0a68b99ecad9\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " Jul 02 17:11:42.453473 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.453441 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwfkl\" (UniqueName: \"kubernetes.io/projected/119393ff-51df-4c90-a0cc-0a68b99ecad9-kube-api-access-rwfkl\") pod \"119393ff-51df-4c90-a0cc-0a68b99ecad9\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " Jul 02 17:11:42.453691 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.453493 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/119393ff-51df-4c90-a0cc-0a68b99ecad9-oidc-ca\") pod \"119393ff-51df-4c90-a0cc-0a68b99ecad9\" (UID: \"119393ff-51df-4c90-a0cc-0a68b99ecad9\") " Jul 02 17:11:42.455642 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.455614 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119393ff-51df-4c90-a0cc-0a68b99ecad9-kube-api-access-rwfkl" (OuterVolumeSpecName: "kube-api-access-rwfkl") pod "119393ff-51df-4c90-a0cc-0a68b99ecad9" (UID: "119393ff-51df-4c90-a0cc-0a68b99ecad9"). InnerVolumeSpecName "kube-api-access-rwfkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:11:42.459385 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.459355 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119393ff-51df-4c90-a0cc-0a68b99ecad9-oidc-ca" (OuterVolumeSpecName: "oidc-ca") pod "119393ff-51df-4c90-a0cc-0a68b99ecad9" (UID: "119393ff-51df-4c90-a0cc-0a68b99ecad9"). InnerVolumeSpecName "oidc-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:11:42.464072 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.464043 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119393ff-51df-4c90-a0cc-0a68b99ecad9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "119393ff-51df-4c90-a0cc-0a68b99ecad9" (UID: "119393ff-51df-4c90-a0cc-0a68b99ecad9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:11:42.554632 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.554598 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119393ff-51df-4c90-a0cc-0a68b99ecad9-tls-cert\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:11:42.554632 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.554622 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwfkl\" (UniqueName: \"kubernetes.io/projected/119393ff-51df-4c90-a0cc-0a68b99ecad9-kube-api-access-rwfkl\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:11:42.554632 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:42.554635 2582 reconciler_common.go:299] "Volume detached for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/119393ff-51df-4c90-a0cc-0a68b99ecad9-oidc-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:11:43.100608 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.100570 2582 generic.go:356] "Generic (PLEG): container finished" podID="119393ff-51df-4c90-a0cc-0a68b99ecad9" containerID="40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1" exitCode=0 Jul 02 17:11:43.101038 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.100619 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cd78889d6-cjph5" Jul 02 17:11:43.101038 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.100658 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cd78889d6-cjph5" event={"ID":"119393ff-51df-4c90-a0cc-0a68b99ecad9","Type":"ContainerDied","Data":"40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1"} Jul 02 17:11:43.101038 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.100707 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cd78889d6-cjph5" event={"ID":"119393ff-51df-4c90-a0cc-0a68b99ecad9","Type":"ContainerDied","Data":"2e954b82a967ba8d4b7acba6e618f5b3153c5492364103e9558a1e74dde9323f"} Jul 02 17:11:43.101038 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.100753 2582 scope.go:117] "RemoveContainer" containerID="40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1" Jul 02 17:11:43.110868 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.110851 2582 scope.go:117] "RemoveContainer" containerID="40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1" Jul 02 17:11:43.111138 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:11:43.111121 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1\": container with ID starting with 40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1 not found: ID does not exist" containerID="40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1" Jul 02 17:11:43.111183 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.111145 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1"} err="failed to get container status \"40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1\": rpc error: code = NotFound desc = could not find container \"40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1\": container with ID starting with 40bc3b5fd51ea2f02523085d9bb3c8bce0d3b6554a6bbc39d29abf2d50d5a6d1 not found: ID does not exist" Jul 02 17:11:43.127090 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.127066 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cd78889d6-cjph5"] Jul 02 17:11:43.135537 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.135508 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cd78889d6-cjph5"] Jul 02 17:11:43.267805 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:43.267773 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="119393ff-51df-4c90-a0cc-0a68b99ecad9" path="/var/lib/kubelet/pods/119393ff-51df-4c90-a0cc-0a68b99ecad9/volumes" Jul 02 17:11:53.253655 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:53.253576 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:11:53.254105 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:11:53.253869 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:12:56.345795 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.345760 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qgbn7"] Jul 02 17:12:56.348284 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.346000 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" podUID="b753e89a-0b0e-42a4-8c73-48b620377dc0" containerName="manager" containerID="cri-o://eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1" gracePeriod=30 Jul 02 17:12:56.589398 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.589371 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:12:56.705716 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.705680 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b753e89a-0b0e-42a4-8c73-48b620377dc0-webhook-cert\") pod \"b753e89a-0b0e-42a4-8c73-48b620377dc0\" (UID: \"b753e89a-0b0e-42a4-8c73-48b620377dc0\") " Jul 02 17:12:56.705716 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.705721 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6crmm\" (UniqueName: \"kubernetes.io/projected/b753e89a-0b0e-42a4-8c73-48b620377dc0-kube-api-access-6crmm\") pod \"b753e89a-0b0e-42a4-8c73-48b620377dc0\" (UID: \"b753e89a-0b0e-42a4-8c73-48b620377dc0\") " Jul 02 17:12:56.707869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.707836 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b753e89a-0b0e-42a4-8c73-48b620377dc0-kube-api-access-6crmm" (OuterVolumeSpecName: "kube-api-access-6crmm") pod "b753e89a-0b0e-42a4-8c73-48b620377dc0" (UID: "b753e89a-0b0e-42a4-8c73-48b620377dc0"). InnerVolumeSpecName "kube-api-access-6crmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:12:56.707992 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.707841 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b753e89a-0b0e-42a4-8c73-48b620377dc0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b753e89a-0b0e-42a4-8c73-48b620377dc0" (UID: "b753e89a-0b0e-42a4-8c73-48b620377dc0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:12:56.807258 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.807223 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b753e89a-0b0e-42a4-8c73-48b620377dc0-webhook-cert\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:12:56.807258 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:56.807253 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6crmm\" (UniqueName: \"kubernetes.io/projected/b753e89a-0b0e-42a4-8c73-48b620377dc0-kube-api-access-6crmm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:12:57.445318 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.445284 2582 generic.go:356] "Generic (PLEG): container finished" podID="b753e89a-0b0e-42a4-8c73-48b620377dc0" containerID="eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1" exitCode=0 Jul 02 17:12:57.445826 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.445334 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" Jul 02 17:12:57.445826 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.445374 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" event={"ID":"b753e89a-0b0e-42a4-8c73-48b620377dc0","Type":"ContainerDied","Data":"eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1"} Jul 02 17:12:57.445826 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.445443 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dfcd6c9c4-qgbn7" event={"ID":"b753e89a-0b0e-42a4-8c73-48b620377dc0","Type":"ContainerDied","Data":"cbbfe5b94fff94a7d0132539fbda973470acdd2c6e5952d13ef3271300794b47"} Jul 02 17:12:57.445826 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.445479 2582 scope.go:117] "RemoveContainer" containerID="eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1" Jul 02 17:12:57.453944 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.453927 2582 scope.go:117] "RemoveContainer" containerID="eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1" Jul 02 17:12:57.454207 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:12:57.454191 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1\": container with ID starting with eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1 not found: ID does not exist" containerID="eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1" Jul 02 17:12:57.454251 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.454214 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1"} err="failed to get container status \"eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1\": rpc error: code = NotFound desc = could not find container \"eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1\": container with ID starting with eb8f083d2c9e8107ab79125e6ce13b3127daf97e805a886fd59f09e01804a4d1 not found: ID does not exist" Jul 02 17:12:57.461141 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.461118 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qgbn7"] Jul 02 17:12:57.465603 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:57.465580 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qgbn7"] Jul 02 17:12:58.266006 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.265974 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qwn6v"] Jul 02 17:12:58.266355 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.266339 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119393ff-51df-4c90-a0cc-0a68b99ecad9" containerName="authorino" Jul 02 17:12:58.266462 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.266359 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="119393ff-51df-4c90-a0cc-0a68b99ecad9" containerName="authorino" Jul 02 17:12:58.266462 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.266379 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b753e89a-0b0e-42a4-8c73-48b620377dc0" containerName="manager" Jul 02 17:12:58.266462 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.266387 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b753e89a-0b0e-42a4-8c73-48b620377dc0" containerName="manager" Jul 02 17:12:58.266610 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.266484 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b753e89a-0b0e-42a4-8c73-48b620377dc0" containerName="manager" Jul 02 17:12:58.266610 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.266499 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="119393ff-51df-4c90-a0cc-0a68b99ecad9" containerName="authorino" Jul 02 17:12:58.270620 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.270601 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:58.273842 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.273824 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 17:12:58.273934 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.273917 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8skzx\"" Jul 02 17:12:58.274013 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.273847 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 17:12:58.279539 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.279517 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qwn6v"] Jul 02 17:12:58.321527 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.321497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d164e70-5e2d-413d-999a-771b4e7b7881-webhook-cert\") pod \"maas-controller-7dfcd6c9c4-qwn6v\" (UID: \"2d164e70-5e2d-413d-999a-771b4e7b7881\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:58.321527 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.321527 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krwq9\" (UniqueName: \"kubernetes.io/projected/2d164e70-5e2d-413d-999a-771b4e7b7881-kube-api-access-krwq9\") pod \"maas-controller-7dfcd6c9c4-qwn6v\" (UID: \"2d164e70-5e2d-413d-999a-771b4e7b7881\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:58.422544 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.422509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d164e70-5e2d-413d-999a-771b4e7b7881-webhook-cert\") pod \"maas-controller-7dfcd6c9c4-qwn6v\" (UID: \"2d164e70-5e2d-413d-999a-771b4e7b7881\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:58.422544 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.422544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krwq9\" (UniqueName: \"kubernetes.io/projected/2d164e70-5e2d-413d-999a-771b4e7b7881-kube-api-access-krwq9\") pod \"maas-controller-7dfcd6c9c4-qwn6v\" (UID: \"2d164e70-5e2d-413d-999a-771b4e7b7881\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:58.425056 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.425036 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d164e70-5e2d-413d-999a-771b4e7b7881-webhook-cert\") pod \"maas-controller-7dfcd6c9c4-qwn6v\" (UID: \"2d164e70-5e2d-413d-999a-771b4e7b7881\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:58.431435 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.431407 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krwq9\" (UniqueName: \"kubernetes.io/projected/2d164e70-5e2d-413d-999a-771b4e7b7881-kube-api-access-krwq9\") pod \"maas-controller-7dfcd6c9c4-qwn6v\" (UID: \"2d164e70-5e2d-413d-999a-771b4e7b7881\") " pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:58.582031 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.581941 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:58.710433 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:58.710409 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dfcd6c9c4-qwn6v"] Jul 02 17:12:58.712328 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:12:58.712293 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d164e70_5e2d_413d_999a_771b4e7b7881.slice/crio-9152ae0196b167a6ea2832aae6e48805605ddc36a12511a05ef2fea2f2b77791 WatchSource:0}: Error finding container 9152ae0196b167a6ea2832aae6e48805605ddc36a12511a05ef2fea2f2b77791: Status 404 returned error can't find the container with id 9152ae0196b167a6ea2832aae6e48805605ddc36a12511a05ef2fea2f2b77791 Jul 02 17:12:59.266256 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:59.266231 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b753e89a-0b0e-42a4-8c73-48b620377dc0" path="/var/lib/kubelet/pods/b753e89a-0b0e-42a4-8c73-48b620377dc0/volumes" Jul 02 17:12:59.456039 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:59.456002 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" event={"ID":"2d164e70-5e2d-413d-999a-771b4e7b7881","Type":"ContainerStarted","Data":"0a635880db854f4fa9bf31d6331cb08d183d981de17f3c7015c8b0df8ae219c5"} Jul 02 17:12:59.456205 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:59.456059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" event={"ID":"2d164e70-5e2d-413d-999a-771b4e7b7881","Type":"ContainerStarted","Data":"9152ae0196b167a6ea2832aae6e48805605ddc36a12511a05ef2fea2f2b77791"} Jul 02 17:12:59.456205 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:59.456089 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:12:59.472665 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:12:59.472619 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" podStartSLOduration=0.920805233 podStartE2EDuration="1.472604655s" totalImagesPullingTime="551.799422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:12:59.471586988 +0000 UTC m=+966.660904021" watchObservedRunningTime="2026-07-02 17:12:59.472604655 +0000 UTC m=+966.661921716" Jul 02 17:13:10.465248 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:13:10.465215 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dfcd6c9c4-qwn6v" Jul 02 17:13:53.292198 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:13:53.292163 2582 scope.go:117] "RemoveContainer" containerID="e4db40fb2eaa2e1186bc8a841ba33cef31fe311fb0995ed776bb1ecdd04e55a4" Jul 02 17:15:00.141749 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.141661 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29716875-hqmcf"] Jul 02 17:15:00.144521 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.144503 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" Jul 02 17:15:00.147150 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.147134 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:15:00.196106 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.196076 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29716875-hqmcf"] Jul 02 17:15:00.249843 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.249814 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4tt\" (UniqueName: \"kubernetes.io/projected/7f94ad41-99b8-4f34-82f9-b884d0016656-kube-api-access-zq4tt\") pod \"maas-api-key-cleanup-29716875-hqmcf\" (UID: \"7f94ad41-99b8-4f34-82f9-b884d0016656\") " pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" Jul 02 17:15:00.350205 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.350173 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4tt\" (UniqueName: \"kubernetes.io/projected/7f94ad41-99b8-4f34-82f9-b884d0016656-kube-api-access-zq4tt\") pod \"maas-api-key-cleanup-29716875-hqmcf\" (UID: \"7f94ad41-99b8-4f34-82f9-b884d0016656\") " pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" Jul 02 17:15:00.358372 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.358349 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4tt\" (UniqueName: \"kubernetes.io/projected/7f94ad41-99b8-4f34-82f9-b884d0016656-kube-api-access-zq4tt\") pod \"maas-api-key-cleanup-29716875-hqmcf\" (UID: \"7f94ad41-99b8-4f34-82f9-b884d0016656\") " pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" Jul 02 17:15:00.454209 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.454132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" Jul 02 17:15:00.788221 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.788198 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29716875-hqmcf"] Jul 02 17:15:00.790566 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:15:00.790538 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f94ad41_99b8_4f34_82f9_b884d0016656.slice/crio-d4e916d1ac10db3712df3030079aa03ab2f4ca705407c49fab0a799a8437b5e2 WatchSource:0}: Error finding container d4e916d1ac10db3712df3030079aa03ab2f4ca705407c49fab0a799a8437b5e2: Status 404 returned error can't find the container with id d4e916d1ac10db3712df3030079aa03ab2f4ca705407c49fab0a799a8437b5e2 Jul 02 17:15:00.792404 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.792330 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 17:15:00.977811 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:00.977768 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" event={"ID":"7f94ad41-99b8-4f34-82f9-b884d0016656","Type":"ContainerStarted","Data":"d4e916d1ac10db3712df3030079aa03ab2f4ca705407c49fab0a799a8437b5e2"} Jul 02 17:15:03.993212 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:03.993118 2582 generic.go:356] "Generic (PLEG): container finished" podID="7f94ad41-99b8-4f34-82f9-b884d0016656" containerID="a7177882d9de81a85d33b28cf957829e851b365bac2c716f844389e9c922370a" exitCode=0 Jul 02 17:15:03.993212 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:03.993164 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" event={"ID":"7f94ad41-99b8-4f34-82f9-b884d0016656","Type":"ContainerDied","Data":"a7177882d9de81a85d33b28cf957829e851b365bac2c716f844389e9c922370a"} Jul 02 17:15:05.133169 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:05.133143 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" Jul 02 17:15:05.191328 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:05.191294 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq4tt\" (UniqueName: \"kubernetes.io/projected/7f94ad41-99b8-4f34-82f9-b884d0016656-kube-api-access-zq4tt\") pod \"7f94ad41-99b8-4f34-82f9-b884d0016656\" (UID: \"7f94ad41-99b8-4f34-82f9-b884d0016656\") " Jul 02 17:15:05.193371 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:05.193340 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f94ad41-99b8-4f34-82f9-b884d0016656-kube-api-access-zq4tt" (OuterVolumeSpecName: "kube-api-access-zq4tt") pod "7f94ad41-99b8-4f34-82f9-b884d0016656" (UID: "7f94ad41-99b8-4f34-82f9-b884d0016656"). InnerVolumeSpecName "kube-api-access-zq4tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:15:05.292864 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:05.292832 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq4tt\" (UniqueName: \"kubernetes.io/projected/7f94ad41-99b8-4f34-82f9-b884d0016656-kube-api-access-zq4tt\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:15:06.002885 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:06.002853 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" Jul 02 17:15:06.002885 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:06.002874 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29716875-hqmcf" event={"ID":"7f94ad41-99b8-4f34-82f9-b884d0016656","Type":"ContainerDied","Data":"d4e916d1ac10db3712df3030079aa03ab2f4ca705407c49fab0a799a8437b5e2"} Jul 02 17:15:06.003089 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:15:06.002916 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e916d1ac10db3712df3030079aa03ab2f4ca705407c49fab0a799a8437b5e2" Jul 02 17:16:53.278307 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:16:53.278278 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:16:53.279334 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:16:53.279310 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:21:53.304747 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:21:53.304702 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:21:53.306284 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:21:53.306262 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:24:53.975842 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:53.975803 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq"] Jul 02 17:24:53.976334 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:53.976318 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f94ad41-99b8-4f34-82f9-b884d0016656" containerName="cleanup" Jul 02 17:24:53.976408 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:53.976336 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f94ad41-99b8-4f34-82f9-b884d0016656" containerName="cleanup" Jul 02 17:24:53.976474 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:53.976461 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f94ad41-99b8-4f34-82f9-b884d0016656" containerName="cleanup" Jul 02 17:24:53.979795 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:53.979775 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:53.982939 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:53.982917 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:24:53.983074 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:53.983057 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-e9de6c51-serving-cert\"" Jul 02 17:24:53.990654 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:53.990517 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq"] Jul 02 17:24:54.032962 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.032929 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.032962 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.032961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2344e-da70-4229-bc81-9b67d4eadc08-openshift-service-ca\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.033146 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.033006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw489\" (UniqueName: \"kubernetes.io/projected/19f2344e-da70-4229-bc81-9b67d4eadc08-kube-api-access-bw489\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.134421 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.134383 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.134421 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.134420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2344e-da70-4229-bc81-9b67d4eadc08-openshift-service-ca\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.134642 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.134456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw489\" (UniqueName: \"kubernetes.io/projected/19f2344e-da70-4229-bc81-9b67d4eadc08-kube-api-access-bw489\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.134642 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:24:54.134543 2582 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-e9de6c51-serving-cert: secret "maas-api-e2e-ait-e9de6c51-serving-cert" not found Jul 02 17:24:54.134642 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:24:54.134627 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls podName:19f2344e-da70-4229-bc81-9b67d4eadc08 nodeName:}" failed. No retries permitted until 2026-07-02 17:24:54.634606981 +0000 UTC m=+1681.823923992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls") pod "maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" (UID: "19f2344e-da70-4229-bc81-9b67d4eadc08") : secret "maas-api-e2e-ait-e9de6c51-serving-cert" not found Jul 02 17:24:54.135172 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.135153 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2344e-da70-4229-bc81-9b67d4eadc08-openshift-service-ca\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.145327 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.145298 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw489\" (UniqueName: \"kubernetes.io/projected/19f2344e-da70-4229-bc81-9b67d4eadc08-kube-api-access-bw489\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.640100 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.640063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.642565 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.642538 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls\") pod \"maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:54.891843 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:54.891755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:55.033100 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:55.033022 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq"] Jul 02 17:24:55.035475 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:24:55.035451 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f2344e_da70_4229_bc81_9b67d4eadc08.slice/crio-50d52fcc6e88bbc63e03403932ea77822730fba5886d769231ae4d75fd91bae6 WatchSource:0}: Error finding container 50d52fcc6e88bbc63e03403932ea77822730fba5886d769231ae4d75fd91bae6: Status 404 returned error can't find the container with id 50d52fcc6e88bbc63e03403932ea77822730fba5886d769231ae4d75fd91bae6 Jul 02 17:24:55.036998 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:55.036978 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 17:24:55.669887 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:55.669846 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" event={"ID":"19f2344e-da70-4229-bc81-9b67d4eadc08","Type":"ContainerStarted","Data":"50d52fcc6e88bbc63e03403932ea77822730fba5886d769231ae4d75fd91bae6"} Jul 02 17:24:57.683589 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:57.683548 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" event={"ID":"19f2344e-da70-4229-bc81-9b67d4eadc08","Type":"ContainerStarted","Data":"7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403"} Jul 02 17:24:57.684015 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:57.683645 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:57.705560 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:57.705502 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" podStartSLOduration=2.490667034 podStartE2EDuration="4.705482951s" totalImagesPullingTime="2.214815917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:24:57.703409009 +0000 UTC m=+1684.892726042" watchObservedRunningTime="2026-07-02 17:24:57.705482951 +0000 UTC m=+1684.894799983" Jul 02 17:24:59.456675 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.456642 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq"] Jul 02 17:24:59.691609 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.691549 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" podUID="19f2344e-da70-4229-bc81-9b67d4eadc08" containerName="maas-api" containerID="cri-o://7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403" gracePeriod=30 Jul 02 17:24:59.935353 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.935328 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:24:59.988405 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.988377 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2344e-da70-4229-bc81-9b67d4eadc08-openshift-service-ca\") pod \"19f2344e-da70-4229-bc81-9b67d4eadc08\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " Jul 02 17:24:59.988569 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.988413 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw489\" (UniqueName: \"kubernetes.io/projected/19f2344e-da70-4229-bc81-9b67d4eadc08-kube-api-access-bw489\") pod \"19f2344e-da70-4229-bc81-9b67d4eadc08\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " Jul 02 17:24:59.988637 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.988563 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls\") pod \"19f2344e-da70-4229-bc81-9b67d4eadc08\" (UID: \"19f2344e-da70-4229-bc81-9b67d4eadc08\") " Jul 02 17:24:59.988839 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.988809 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f2344e-da70-4229-bc81-9b67d4eadc08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19f2344e-da70-4229-bc81-9b67d4eadc08" (UID: "19f2344e-da70-4229-bc81-9b67d4eadc08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:24:59.990580 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.990561 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19f2344e-da70-4229-bc81-9b67d4eadc08" (UID: "19f2344e-da70-4229-bc81-9b67d4eadc08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:24:59.990682 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:24:59.990664 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f2344e-da70-4229-bc81-9b67d4eadc08-kube-api-access-bw489" (OuterVolumeSpecName: "kube-api-access-bw489") pod "19f2344e-da70-4229-bc81-9b67d4eadc08" (UID: "19f2344e-da70-4229-bc81-9b67d4eadc08"). InnerVolumeSpecName "kube-api-access-bw489". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:25:00.089960 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.089872 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19f2344e-da70-4229-bc81-9b67d4eadc08-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:00.089960 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.089903 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2344e-da70-4229-bc81-9b67d4eadc08-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:00.089960 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.089918 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw489\" (UniqueName: \"kubernetes.io/projected/19f2344e-da70-4229-bc81-9b67d4eadc08-kube-api-access-bw489\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:00.698626 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.698592 2582 generic.go:356] "Generic (PLEG): container finished" podID="19f2344e-da70-4229-bc81-9b67d4eadc08" containerID="7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403" exitCode=0 Jul 02 17:25:00.699032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.698640 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" Jul 02 17:25:00.699032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.698677 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" event={"ID":"19f2344e-da70-4229-bc81-9b67d4eadc08","Type":"ContainerDied","Data":"7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403"} Jul 02 17:25:00.699032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.698755 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq" event={"ID":"19f2344e-da70-4229-bc81-9b67d4eadc08","Type":"ContainerDied","Data":"50d52fcc6e88bbc63e03403932ea77822730fba5886d769231ae4d75fd91bae6"} Jul 02 17:25:00.699032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.698779 2582 scope.go:117] "RemoveContainer" containerID="7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403" Jul 02 17:25:00.707515 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.707493 2582 scope.go:117] "RemoveContainer" containerID="7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403" Jul 02 17:25:00.707837 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:25:00.707812 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403\": container with ID starting with 7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403 not found: ID does not exist" containerID="7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403" Jul 02 17:25:00.707934 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.707845 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403"} err="failed to get container status \"7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403\": rpc error: code = NotFound desc = could not find container \"7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403\": container with ID starting with 7b1f562b58d5c5ffa5ae6313e060b639bc37f9b851b08a0ad9a63835faf34403 not found: ID does not exist" Jul 02 17:25:00.721396 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.721363 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq"] Jul 02 17:25:00.724765 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:00.724741 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-e9de6c51-6c5654f68b-wjjkq"] Jul 02 17:25:01.265993 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:01.265962 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19f2344e-da70-4229-bc81-9b67d4eadc08" path="/var/lib/kubelet/pods/19f2344e-da70-4229-bc81-9b67d4eadc08/volumes" Jul 02 17:25:06.549216 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.549188 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9"] Jul 02 17:25:06.549606 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.549526 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19f2344e-da70-4229-bc81-9b67d4eadc08" containerName="maas-api" Jul 02 17:25:06.549606 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.549542 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="19f2344e-da70-4229-bc81-9b67d4eadc08" containerName="maas-api" Jul 02 17:25:06.549684 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.549611 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="19f2344e-da70-4229-bc81-9b67d4eadc08" containerName="maas-api" Jul 02 17:25:06.554109 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.554085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.556534 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.556510 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-d4601268-serving-cert\"" Jul 02 17:25:06.557232 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.557214 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:25:06.567217 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.567193 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9"] Jul 02 17:25:06.650069 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.650032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zpv\" (UniqueName: \"kubernetes.io/projected/c8db2826-470d-4bf7-8910-f6c7745ea292-kube-api-access-r5zpv\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.650249 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.650079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db2826-470d-4bf7-8910-f6c7745ea292-openshift-service-ca\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.650249 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.650182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db2826-470d-4bf7-8910-f6c7745ea292-maas-api-tls\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.750989 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.750950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zpv\" (UniqueName: \"kubernetes.io/projected/c8db2826-470d-4bf7-8910-f6c7745ea292-kube-api-access-r5zpv\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.750989 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.750992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db2826-470d-4bf7-8910-f6c7745ea292-openshift-service-ca\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.751201 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.751042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db2826-470d-4bf7-8910-f6c7745ea292-maas-api-tls\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.751881 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.751859 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db2826-470d-4bf7-8910-f6c7745ea292-openshift-service-ca\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.753547 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.753526 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db2826-470d-4bf7-8910-f6c7745ea292-maas-api-tls\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.759687 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.759665 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zpv\" (UniqueName: \"kubernetes.io/projected/c8db2826-470d-4bf7-8910-f6c7745ea292-kube-api-access-r5zpv\") pod \"maas-api-e2e-ait-d4601268-565899d867-mmqk9\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:06.866982 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:06.866889 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:07.006447 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:07.006413 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9"] Jul 02 17:25:07.011255 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:25:07.011222 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8db2826_470d_4bf7_8910_f6c7745ea292.slice/crio-20332b5a76cbca38a93283d0f7346cca5898daa4936f47d17a489b6e11ea2e25 WatchSource:0}: Error finding container 20332b5a76cbca38a93283d0f7346cca5898daa4936f47d17a489b6e11ea2e25: Status 404 returned error can't find the container with id 20332b5a76cbca38a93283d0f7346cca5898daa4936f47d17a489b6e11ea2e25 Jul 02 17:25:07.738987 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:07.738953 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" event={"ID":"c8db2826-470d-4bf7-8910-f6c7745ea292","Type":"ContainerStarted","Data":"c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e"} Jul 02 17:25:07.739463 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:07.739009 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" event={"ID":"c8db2826-470d-4bf7-8910-f6c7745ea292","Type":"ContainerStarted","Data":"20332b5a76cbca38a93283d0f7346cca5898daa4936f47d17a489b6e11ea2e25"} Jul 02 17:25:07.739463 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:07.739081 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:07.758951 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:07.758902 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" podStartSLOduration=1.409575635 podStartE2EDuration="1.758886272s" totalImagesPullingTime="349.310637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:25:07.757251313 +0000 UTC m=+1694.946568345" watchObservedRunningTime="2026-07-02 17:25:07.758886272 +0000 UTC m=+1694.948203302" Jul 02 17:25:11.978145 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:11.978103 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9"] Jul 02 17:25:11.978495 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:11.978314 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" podUID="c8db2826-470d-4bf7-8910-f6c7745ea292" containerName="maas-api" containerID="cri-o://c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e" gracePeriod=30 Jul 02 17:25:12.221615 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.221593 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:12.302422 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.302338 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db2826-470d-4bf7-8910-f6c7745ea292-maas-api-tls\") pod \"c8db2826-470d-4bf7-8910-f6c7745ea292\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " Jul 02 17:25:12.302422 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.302391 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db2826-470d-4bf7-8910-f6c7745ea292-openshift-service-ca\") pod \"c8db2826-470d-4bf7-8910-f6c7745ea292\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " Jul 02 17:25:12.302637 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.302428 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5zpv\" (UniqueName: \"kubernetes.io/projected/c8db2826-470d-4bf7-8910-f6c7745ea292-kube-api-access-r5zpv\") pod \"c8db2826-470d-4bf7-8910-f6c7745ea292\" (UID: \"c8db2826-470d-4bf7-8910-f6c7745ea292\") " Jul 02 17:25:12.302793 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.302762 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8db2826-470d-4bf7-8910-f6c7745ea292-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8db2826-470d-4bf7-8910-f6c7745ea292" (UID: "c8db2826-470d-4bf7-8910-f6c7745ea292"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:25:12.304384 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.304362 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8db2826-470d-4bf7-8910-f6c7745ea292-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8db2826-470d-4bf7-8910-f6c7745ea292" (UID: "c8db2826-470d-4bf7-8910-f6c7745ea292"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:25:12.304563 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.304548 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8db2826-470d-4bf7-8910-f6c7745ea292-kube-api-access-r5zpv" (OuterVolumeSpecName: "kube-api-access-r5zpv") pod "c8db2826-470d-4bf7-8910-f6c7745ea292" (UID: "c8db2826-470d-4bf7-8910-f6c7745ea292"). InnerVolumeSpecName "kube-api-access-r5zpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:25:12.403056 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.403022 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db2826-470d-4bf7-8910-f6c7745ea292-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:12.403056 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.403049 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zpv\" (UniqueName: \"kubernetes.io/projected/c8db2826-470d-4bf7-8910-f6c7745ea292-kube-api-access-r5zpv\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:12.403056 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.403060 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db2826-470d-4bf7-8910-f6c7745ea292-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:12.762192 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.762154 2582 generic.go:356] "Generic (PLEG): container finished" podID="c8db2826-470d-4bf7-8910-f6c7745ea292" containerID="c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e" exitCode=0 Jul 02 17:25:12.762192 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.762195 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" Jul 02 17:25:12.762392 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.762226 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" event={"ID":"c8db2826-470d-4bf7-8910-f6c7745ea292","Type":"ContainerDied","Data":"c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e"} Jul 02 17:25:12.762392 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.762279 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9" event={"ID":"c8db2826-470d-4bf7-8910-f6c7745ea292","Type":"ContainerDied","Data":"20332b5a76cbca38a93283d0f7346cca5898daa4936f47d17a489b6e11ea2e25"} Jul 02 17:25:12.762392 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.762301 2582 scope.go:117] "RemoveContainer" containerID="c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e" Jul 02 17:25:12.771300 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.771280 2582 scope.go:117] "RemoveContainer" containerID="c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e" Jul 02 17:25:12.771556 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:25:12.771538 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e\": container with ID starting with c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e not found: ID does not exist" containerID="c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e" Jul 02 17:25:12.771604 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.771565 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e"} err="failed to get container status \"c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e\": rpc error: code = NotFound desc = could not find container \"c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e\": container with ID starting with c77230ffae9cde160562e960f1df1fd108c48f81a3ac367d0a6baeb50e38106e not found: ID does not exist" Jul 02 17:25:12.782639 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.782616 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9"] Jul 02 17:25:12.786436 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:12.786413 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-d4601268-565899d867-mmqk9"] Jul 02 17:25:13.267044 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:13.267016 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8db2826-470d-4bf7-8910-f6c7745ea292" path="/var/lib/kubelet/pods/c8db2826-470d-4bf7-8910-f6c7745ea292/volumes" Jul 02 17:25:25.364114 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.364081 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z"] Jul 02 17:25:25.364622 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.364422 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8db2826-470d-4bf7-8910-f6c7745ea292" containerName="maas-api" Jul 02 17:25:25.364622 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.364432 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8db2826-470d-4bf7-8910-f6c7745ea292" containerName="maas-api" Jul 02 17:25:25.364622 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.364499 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8db2826-470d-4bf7-8910-f6c7745ea292" containerName="maas-api" Jul 02 17:25:25.367222 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.367203 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.369575 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.369551 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-57f9f126-serving-cert\"" Jul 02 17:25:25.369924 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.369906 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:25:25.378810 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.378787 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z"] Jul 02 17:25:25.414495 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.414466 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0742841-907c-4574-bb62-4c25a1fc10bd-openshift-service-ca\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.414698 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.414524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0742841-907c-4574-bb62-4c25a1fc10bd-maas-api-tls\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.414698 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.414581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hpm\" (UniqueName: \"kubernetes.io/projected/b0742841-907c-4574-bb62-4c25a1fc10bd-kube-api-access-m7hpm\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.515094 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.515061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0742841-907c-4574-bb62-4c25a1fc10bd-maas-api-tls\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.515266 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.515101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hpm\" (UniqueName: \"kubernetes.io/projected/b0742841-907c-4574-bb62-4c25a1fc10bd-kube-api-access-m7hpm\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.515266 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.515148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0742841-907c-4574-bb62-4c25a1fc10bd-openshift-service-ca\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.515782 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.515764 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0742841-907c-4574-bb62-4c25a1fc10bd-openshift-service-ca\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.517430 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.517412 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0742841-907c-4574-bb62-4c25a1fc10bd-maas-api-tls\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.526076 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.526042 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hpm\" (UniqueName: \"kubernetes.io/projected/b0742841-907c-4574-bb62-4c25a1fc10bd-kube-api-access-m7hpm\") pod \"maas-api-e2e-derive-57f9f126-554c95476f-qzx5z\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.678855 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.678817 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:25.815283 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:25.815255 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z"] Jul 02 17:25:25.817355 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:25:25.817314 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0742841_907c_4574_bb62_4c25a1fc10bd.slice/crio-33dd2f127da20f6d9587f459aa2ef5a1e60dfc234dbd223d1948cc48e2926787 WatchSource:0}: Error finding container 33dd2f127da20f6d9587f459aa2ef5a1e60dfc234dbd223d1948cc48e2926787: Status 404 returned error can't find the container with id 33dd2f127da20f6d9587f459aa2ef5a1e60dfc234dbd223d1948cc48e2926787 Jul 02 17:25:26.828228 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:26.828186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" event={"ID":"b0742841-907c-4574-bb62-4c25a1fc10bd","Type":"ContainerStarted","Data":"3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486"} Jul 02 17:25:26.828600 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:26.828254 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:26.828600 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:26.828280 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" event={"ID":"b0742841-907c-4574-bb62-4c25a1fc10bd","Type":"ContainerStarted","Data":"33dd2f127da20f6d9587f459aa2ef5a1e60dfc234dbd223d1948cc48e2926787"} Jul 02 17:25:26.847466 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:26.847419 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" podStartSLOduration=1.425466262 podStartE2EDuration="1.847403174s" totalImagesPullingTime="421.936912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:25:26.845390856 +0000 UTC m=+1714.034707889" watchObservedRunningTime="2026-07-02 17:25:26.847403174 +0000 UTC m=+1714.036720204" Jul 02 17:25:29.773785 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:29.773744 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z"] Jul 02 17:25:29.774193 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:29.773964 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" podUID="b0742841-907c-4574-bb62-4c25a1fc10bd" containerName="maas-api" containerID="cri-o://3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486" gracePeriod=30 Jul 02 17:25:30.012562 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.012537 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:30.048033 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.047967 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0742841-907c-4574-bb62-4c25a1fc10bd-maas-api-tls\") pod \"b0742841-907c-4574-bb62-4c25a1fc10bd\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " Jul 02 17:25:30.048149 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.048037 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7hpm\" (UniqueName: \"kubernetes.io/projected/b0742841-907c-4574-bb62-4c25a1fc10bd-kube-api-access-m7hpm\") pod \"b0742841-907c-4574-bb62-4c25a1fc10bd\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " Jul 02 17:25:30.048149 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.048095 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0742841-907c-4574-bb62-4c25a1fc10bd-openshift-service-ca\") pod \"b0742841-907c-4574-bb62-4c25a1fc10bd\" (UID: \"b0742841-907c-4574-bb62-4c25a1fc10bd\") " Jul 02 17:25:30.048466 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.048439 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0742841-907c-4574-bb62-4c25a1fc10bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0742841-907c-4574-bb62-4c25a1fc10bd" (UID: "b0742841-907c-4574-bb62-4c25a1fc10bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:25:30.050131 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.050113 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0742841-907c-4574-bb62-4c25a1fc10bd-kube-api-access-m7hpm" (OuterVolumeSpecName: "kube-api-access-m7hpm") pod "b0742841-907c-4574-bb62-4c25a1fc10bd" (UID: "b0742841-907c-4574-bb62-4c25a1fc10bd"). InnerVolumeSpecName "kube-api-access-m7hpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:25:30.050211 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.050193 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0742841-907c-4574-bb62-4c25a1fc10bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0742841-907c-4574-bb62-4c25a1fc10bd" (UID: "b0742841-907c-4574-bb62-4c25a1fc10bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:25:30.148595 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.148563 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0742841-907c-4574-bb62-4c25a1fc10bd-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:30.148595 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.148590 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7hpm\" (UniqueName: \"kubernetes.io/projected/b0742841-907c-4574-bb62-4c25a1fc10bd-kube-api-access-m7hpm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:30.148595 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.148600 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0742841-907c-4574-bb62-4c25a1fc10bd-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:25:30.848402 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.848364 2582 generic.go:356] "Generic (PLEG): container finished" podID="b0742841-907c-4574-bb62-4c25a1fc10bd" containerID="3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486" exitCode=0 Jul 02 17:25:30.848890 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.848407 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" Jul 02 17:25:30.848890 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.848447 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" event={"ID":"b0742841-907c-4574-bb62-4c25a1fc10bd","Type":"ContainerDied","Data":"3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486"} Jul 02 17:25:30.848890 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.848501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z" event={"ID":"b0742841-907c-4574-bb62-4c25a1fc10bd","Type":"ContainerDied","Data":"33dd2f127da20f6d9587f459aa2ef5a1e60dfc234dbd223d1948cc48e2926787"} Jul 02 17:25:30.848890 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.848523 2582 scope.go:117] "RemoveContainer" containerID="3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486" Jul 02 17:25:30.857454 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.857435 2582 scope.go:117] "RemoveContainer" containerID="3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486" Jul 02 17:25:30.857705 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:25:30.857688 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486\": container with ID starting with 3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486 not found: ID does not exist" containerID="3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486" Jul 02 17:25:30.857811 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.857714 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486"} err="failed to get container status \"3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486\": rpc error: code = NotFound desc = could not find container \"3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486\": container with ID starting with 3c6ac4f6008165e759c4226959ce52d5476ba98721a2d2334a1cfa4eacea2486 not found: ID does not exist" Jul 02 17:25:30.869638 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.869611 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z"] Jul 02 17:25:30.871560 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:30.871537 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-57f9f126-554c95476f-qzx5z"] Jul 02 17:25:31.266817 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:25:31.266776 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0742841-907c-4574-bb62-4c25a1fc10bd" path="/var/lib/kubelet/pods/b0742841-907c-4574-bb62-4c25a1fc10bd/volumes" Jul 02 17:26:53.329462 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:26:53.329435 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:26:53.332528 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:26:53.332508 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:27:22.217210 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.217178 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr"] Jul 02 17:27:22.217666 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.217508 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0742841-907c-4574-bb62-4c25a1fc10bd" containerName="maas-api" Jul 02 17:27:22.217666 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.217518 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0742841-907c-4574-bb62-4c25a1fc10bd" containerName="maas-api" Jul 02 17:27:22.217666 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.217582 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0742841-907c-4574-bb62-4c25a1fc10bd" containerName="maas-api" Jul 02 17:27:22.220602 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.220586 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.223158 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.223136 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:27:22.223356 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.223336 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-33ec67fa-serving-cert\"" Jul 02 17:27:22.230897 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.230869 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr"] Jul 02 17:27:22.324865 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.324830 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmkcz\" (UniqueName: \"kubernetes.io/projected/7c643100-5518-4217-838f-c4f589328ac3-kube-api-access-kmkcz\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.324865 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.324867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.325059 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.324930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c643100-5518-4217-838f-c4f589328ac3-openshift-service-ca\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.425799 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.425753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c643100-5518-4217-838f-c4f589328ac3-openshift-service-ca\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.425965 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.425859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmkcz\" (UniqueName: \"kubernetes.io/projected/7c643100-5518-4217-838f-c4f589328ac3-kube-api-access-kmkcz\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.425965 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.425881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.426037 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:27:22.425981 2582 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-33ec67fa-serving-cert: secret "maas-api-e2e-mt-33ec67fa-serving-cert" not found Jul 02 17:27:22.426071 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:27:22.426056 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls podName:7c643100-5518-4217-838f-c4f589328ac3 nodeName:}" failed. No retries permitted until 2026-07-02 17:27:22.926034332 +0000 UTC m=+1830.115351351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls") pod "maas-api-e2e-mt-33ec67fa-6547577775-blzzr" (UID: "7c643100-5518-4217-838f-c4f589328ac3") : secret "maas-api-e2e-mt-33ec67fa-serving-cert" not found Jul 02 17:27:22.426417 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.426399 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c643100-5518-4217-838f-c4f589328ac3-openshift-service-ca\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.434573 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.434549 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmkcz\" (UniqueName: \"kubernetes.io/projected/7c643100-5518-4217-838f-c4f589328ac3-kube-api-access-kmkcz\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.929849 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.929804 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:22.932253 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:22.932226 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls\") pod \"maas-api-e2e-mt-33ec67fa-6547577775-blzzr\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:23.132392 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:23.132343 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:23.266536 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:23.266512 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr"] Jul 02 17:27:23.268361 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:27:23.268334 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c643100_5518_4217_838f_c4f589328ac3.slice/crio-be21ec739e009fbe693ac51dc44fdc33673203660f1b4f714ef5bb11fb837320 WatchSource:0}: Error finding container be21ec739e009fbe693ac51dc44fdc33673203660f1b4f714ef5bb11fb837320: Status 404 returned error can't find the container with id be21ec739e009fbe693ac51dc44fdc33673203660f1b4f714ef5bb11fb837320 Jul 02 17:27:23.350876 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:23.350841 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" event={"ID":"7c643100-5518-4217-838f-c4f589328ac3","Type":"ContainerStarted","Data":"be21ec739e009fbe693ac51dc44fdc33673203660f1b4f714ef5bb11fb837320"} Jul 02 17:27:24.356715 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:24.356683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" event={"ID":"7c643100-5518-4217-838f-c4f589328ac3","Type":"ContainerStarted","Data":"80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be"} Jul 02 17:27:24.357155 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:24.356815 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:24.373171 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:24.373125 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" podStartSLOduration=1.851384966 podStartE2EDuration="2.37310914s" totalImagesPullingTime="521.724174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:27:24.372696738 +0000 UTC m=+1831.562013785" watchObservedRunningTime="2026-07-02 17:27:24.37310914 +0000 UTC m=+1831.562426171" Jul 02 17:27:30.366536 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:30.366505 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:34.092158 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.092126 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr"] Jul 02 17:27:34.092574 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.092331 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" podUID="7c643100-5518-4217-838f-c4f589328ac3" containerName="maas-api" containerID="cri-o://80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be" gracePeriod=30 Jul 02 17:27:34.340274 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.340251 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:34.405857 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.405823 2582 generic.go:356] "Generic (PLEG): container finished" podID="7c643100-5518-4217-838f-c4f589328ac3" containerID="80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be" exitCode=0 Jul 02 17:27:34.406020 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.405870 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" Jul 02 17:27:34.406020 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.405894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" event={"ID":"7c643100-5518-4217-838f-c4f589328ac3","Type":"ContainerDied","Data":"80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be"} Jul 02 17:27:34.406020 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.405946 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr" event={"ID":"7c643100-5518-4217-838f-c4f589328ac3","Type":"ContainerDied","Data":"be21ec739e009fbe693ac51dc44fdc33673203660f1b4f714ef5bb11fb837320"} Jul 02 17:27:34.406020 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.405968 2582 scope.go:117] "RemoveContainer" containerID="80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be" Jul 02 17:27:34.413954 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.413939 2582 scope.go:117] "RemoveContainer" containerID="80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be" Jul 02 17:27:34.414186 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:27:34.414169 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be\": container with ID starting with 80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be not found: ID does not exist" containerID="80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be" Jul 02 17:27:34.414230 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.414195 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be"} err="failed to get container status \"80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be\": rpc error: code = NotFound desc = could not find container \"80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be\": container with ID starting with 80b7fdb584be63f67cb01d1ae44be035cded7bb1f1b442f8fba4cd8f695d89be not found: ID does not exist" Jul 02 17:27:34.432106 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.432079 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls\") pod \"7c643100-5518-4217-838f-c4f589328ac3\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " Jul 02 17:27:34.432237 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.432117 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmkcz\" (UniqueName: \"kubernetes.io/projected/7c643100-5518-4217-838f-c4f589328ac3-kube-api-access-kmkcz\") pod \"7c643100-5518-4217-838f-c4f589328ac3\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " Jul 02 17:27:34.432237 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.432199 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c643100-5518-4217-838f-c4f589328ac3-openshift-service-ca\") pod \"7c643100-5518-4217-838f-c4f589328ac3\" (UID: \"7c643100-5518-4217-838f-c4f589328ac3\") " Jul 02 17:27:34.432584 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.432562 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c643100-5518-4217-838f-c4f589328ac3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c643100-5518-4217-838f-c4f589328ac3" (UID: "7c643100-5518-4217-838f-c4f589328ac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:27:34.434176 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.434150 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c643100-5518-4217-838f-c4f589328ac3" (UID: "7c643100-5518-4217-838f-c4f589328ac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:27:34.434267 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.434182 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c643100-5518-4217-838f-c4f589328ac3-kube-api-access-kmkcz" (OuterVolumeSpecName: "kube-api-access-kmkcz") pod "7c643100-5518-4217-838f-c4f589328ac3" (UID: "7c643100-5518-4217-838f-c4f589328ac3"). InnerVolumeSpecName "kube-api-access-kmkcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:27:34.532929 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.532889 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c643100-5518-4217-838f-c4f589328ac3-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:27:34.532929 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.532924 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c643100-5518-4217-838f-c4f589328ac3-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:27:34.532929 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.532933 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmkcz\" (UniqueName: \"kubernetes.io/projected/7c643100-5518-4217-838f-c4f589328ac3-kube-api-access-kmkcz\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:27:34.727014 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.726976 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr"] Jul 02 17:27:34.731206 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:34.731179 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-33ec67fa-6547577775-blzzr"] Jul 02 17:27:35.265877 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:27:35.265836 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c643100-5518-4217-838f-c4f589328ac3" path="/var/lib/kubelet/pods/7c643100-5518-4217-838f-c4f589328ac3/volumes" Jul 02 17:28:03.403255 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.403219 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8"] Jul 02 17:28:03.403740 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.403652 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c643100-5518-4217-838f-c4f589328ac3" containerName="maas-api" Jul 02 17:28:03.403740 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.403671 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c643100-5518-4217-838f-c4f589328ac3" containerName="maas-api" Jul 02 17:28:03.403846 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.403831 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c643100-5518-4217-838f-c4f589328ac3" containerName="maas-api" Jul 02 17:28:03.406693 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.406675 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:03.409156 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.409137 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:28:03.409344 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.409326 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-cfcf23f1-serving-cert\"" Jul 02 17:28:03.416850 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.416826 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8"] Jul 02 17:28:03.475655 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.475622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e61530-aaff-4147-826d-80c4e0ae5a26-openshift-service-ca\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:03.475880 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.475678 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:03.475880 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.475705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8gb\" (UniqueName: \"kubernetes.io/projected/12e61530-aaff-4147-826d-80c4e0ae5a26-kube-api-access-lf8gb\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:03.576506 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.576468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e61530-aaff-4147-826d-80c4e0ae5a26-openshift-service-ca\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:03.576757 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.576565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:03.576757 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.576593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8gb\" (UniqueName: \"kubernetes.io/projected/12e61530-aaff-4147-826d-80c4e0ae5a26-kube-api-access-lf8gb\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:03.576757 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:28:03.576693 2582 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-cfcf23f1-serving-cert: secret "maas-api-e2e-mt-cfcf23f1-serving-cert" not found Jul 02 17:28:03.576922 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:28:03.576786 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls podName:12e61530-aaff-4147-826d-80c4e0ae5a26 nodeName:}" failed. No retries permitted until 2026-07-02 17:28:04.076767178 +0000 UTC m=+1871.266084191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls") pod "maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" (UID: "12e61530-aaff-4147-826d-80c4e0ae5a26") : secret "maas-api-e2e-mt-cfcf23f1-serving-cert" not found Jul 02 17:28:03.577249 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.577230 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e61530-aaff-4147-826d-80c4e0ae5a26-openshift-service-ca\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:03.584668 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:03.584641 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8gb\" (UniqueName: \"kubernetes.io/projected/12e61530-aaff-4147-826d-80c4e0ae5a26-kube-api-access-lf8gb\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:04.081090 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:04.081052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:04.083488 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:04.083468 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls\") pod \"maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:04.320020 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:04.319987 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:04.454072 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:04.453918 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8"] Jul 02 17:28:04.456459 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:28:04.456433 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e61530_aaff_4147_826d_80c4e0ae5a26.slice/crio-2af1ddad9f77f24f0dbfa9feb8995fce153c2df645b671f28c63117d1a888c44 WatchSource:0}: Error finding container 2af1ddad9f77f24f0dbfa9feb8995fce153c2df645b671f28c63117d1a888c44: Status 404 returned error can't find the container with id 2af1ddad9f77f24f0dbfa9feb8995fce153c2df645b671f28c63117d1a888c44 Jul 02 17:28:04.539331 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:04.539289 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" event={"ID":"12e61530-aaff-4147-826d-80c4e0ae5a26","Type":"ContainerStarted","Data":"2af1ddad9f77f24f0dbfa9feb8995fce153c2df645b671f28c63117d1a888c44"} Jul 02 17:28:05.545740 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:05.545691 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" event={"ID":"12e61530-aaff-4147-826d-80c4e0ae5a26","Type":"ContainerStarted","Data":"80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48"} Jul 02 17:28:05.546126 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:05.545790 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:05.562942 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:05.562897 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" podStartSLOduration=2.160078189 podStartE2EDuration="2.562880578s" totalImagesPullingTime="402.802389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:28:05.561309083 +0000 UTC m=+1872.750626114" watchObservedRunningTime="2026-07-02 17:28:05.562880578 +0000 UTC m=+1872.752197609" Jul 02 17:28:11.555340 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:11.555309 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:19.957804 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:19.957685 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm"] Jul 02 17:28:19.961119 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:19.961098 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:19.963460 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:19.963442 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cfcf23f1\"/\"kube-root-ca.crt\"" Jul 02 17:28:19.964293 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:19.964265 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cfcf23f1\"/\"openshift-service-ca.crt\"" Jul 02 17:28:19.964374 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:19.964346 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cfcf23f1\"/\"e2e-lifecycle-model-cfcf23f1-kserve-self-signed-certs\"" Jul 02 17:28:19.964420 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:19.964369 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cfcf23f1\"/\"default-dockercfg-2dqkd\"" Jul 02 17:28:19.973847 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:19.973824 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm"] Jul 02 17:28:20.019959 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.019926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsvm\" (UniqueName: \"kubernetes.io/projected/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-kube-api-access-gnsvm\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.020096 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.019973 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tls-certs\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.020096 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.020064 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-model-cache\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.020096 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.020095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tmp-dir\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.020216 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.020113 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-dshm\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.020216 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.020141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-home\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.120939 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.120899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsvm\" (UniqueName: \"kubernetes.io/projected/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-kube-api-access-gnsvm\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.121138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.120971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tls-certs\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.121138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.121042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-model-cache\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.121273 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.121156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tmp-dir\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.121273 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.121192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-dshm\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.121384 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.121277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-home\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.121564 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.121541 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tmp-dir\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.121684 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.121577 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-model-cache\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.121684 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.121616 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-home\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.123950 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.123923 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-dshm\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.124202 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.124178 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tls-certs\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.130063 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.130032 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsvm\" (UniqueName: \"kubernetes.io/projected/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-kube-api-access-gnsvm\") pod \"e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.270322 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.270234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:20.404429 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.404402 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm"] Jul 02 17:28:20.406682 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:28:20.406650 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc86898f1_7cbf_49b7_b564_d97be1f3cdf1.slice/crio-9106b4b9c7588d092beab1bbd9889761144a0cb183cc3ee0339f6b4be36757c9 WatchSource:0}: Error finding container 9106b4b9c7588d092beab1bbd9889761144a0cb183cc3ee0339f6b4be36757c9: Status 404 returned error can't find the container with id 9106b4b9c7588d092beab1bbd9889761144a0cb183cc3ee0339f6b4be36757c9 Jul 02 17:28:20.617490 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.617403 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" event={"ID":"c86898f1-7cbf-49b7-b564-d97be1f3cdf1","Type":"ContainerStarted","Data":"94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e"} Jul 02 17:28:20.617490 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.617445 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" event={"ID":"c86898f1-7cbf-49b7-b564-d97be1f3cdf1","Type":"ContainerStarted","Data":"9106b4b9c7588d092beab1bbd9889761144a0cb183cc3ee0339f6b4be36757c9"} Jul 02 17:28:20.635077 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:20.635024 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" podStartSLOduration=1.6350094899999998 podStartE2EDuration="1.63500949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:28:20.632823146 +0000 UTC m=+1887.822140176" watchObservedRunningTime="2026-07-02 17:28:20.63500949 +0000 UTC m=+1887.824326521" Jul 02 17:28:30.271284 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:30.271246 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:30.271691 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:30.271299 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:30.283593 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:30.283565 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:30.677790 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:30.677758 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:28:40.641917 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.641885 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8"] Jul 02 17:28:40.642412 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.642098 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" podUID="12e61530-aaff-4147-826d-80c4e0ae5a26" containerName="maas-api" containerID="cri-o://80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48" gracePeriod=30 Jul 02 17:28:40.887592 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.887569 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:40.908598 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.908546 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf8gb\" (UniqueName: \"kubernetes.io/projected/12e61530-aaff-4147-826d-80c4e0ae5a26-kube-api-access-lf8gb\") pod \"12e61530-aaff-4147-826d-80c4e0ae5a26\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " Jul 02 17:28:40.908598 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.908585 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e61530-aaff-4147-826d-80c4e0ae5a26-openshift-service-ca\") pod \"12e61530-aaff-4147-826d-80c4e0ae5a26\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " Jul 02 17:28:40.908743 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.908633 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls\") pod \"12e61530-aaff-4147-826d-80c4e0ae5a26\" (UID: \"12e61530-aaff-4147-826d-80c4e0ae5a26\") " Jul 02 17:28:40.909001 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.908980 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e61530-aaff-4147-826d-80c4e0ae5a26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12e61530-aaff-4147-826d-80c4e0ae5a26" (UID: "12e61530-aaff-4147-826d-80c4e0ae5a26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:28:40.910599 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.910572 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e61530-aaff-4147-826d-80c4e0ae5a26-kube-api-access-lf8gb" (OuterVolumeSpecName: "kube-api-access-lf8gb") pod "12e61530-aaff-4147-826d-80c4e0ae5a26" (UID: "12e61530-aaff-4147-826d-80c4e0ae5a26"). InnerVolumeSpecName "kube-api-access-lf8gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:28:40.910762 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:40.910738 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12e61530-aaff-4147-826d-80c4e0ae5a26" (UID: "12e61530-aaff-4147-826d-80c4e0ae5a26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:28:41.009345 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.009310 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf8gb\" (UniqueName: \"kubernetes.io/projected/12e61530-aaff-4147-826d-80c4e0ae5a26-kube-api-access-lf8gb\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:28:41.009345 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.009338 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e61530-aaff-4147-826d-80c4e0ae5a26-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:28:41.009345 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.009352 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e61530-aaff-4147-826d-80c4e0ae5a26-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:28:41.715443 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.715403 2582 generic.go:356] "Generic (PLEG): container finished" podID="12e61530-aaff-4147-826d-80c4e0ae5a26" containerID="80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48" exitCode=0 Jul 02 17:28:41.715872 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.715464 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" Jul 02 17:28:41.715872 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.715483 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" event={"ID":"12e61530-aaff-4147-826d-80c4e0ae5a26","Type":"ContainerDied","Data":"80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48"} Jul 02 17:28:41.715872 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.715562 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8" event={"ID":"12e61530-aaff-4147-826d-80c4e0ae5a26","Type":"ContainerDied","Data":"2af1ddad9f77f24f0dbfa9feb8995fce153c2df645b671f28c63117d1a888c44"} Jul 02 17:28:41.715872 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.715588 2582 scope.go:117] "RemoveContainer" containerID="80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48" Jul 02 17:28:41.724234 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.724217 2582 scope.go:117] "RemoveContainer" containerID="80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48" Jul 02 17:28:41.724487 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:28:41.724467 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48\": container with ID starting with 80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48 not found: ID does not exist" containerID="80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48" Jul 02 17:28:41.724550 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.724493 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48"} err="failed to get container status \"80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48\": rpc error: code = NotFound desc = could not find container \"80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48\": container with ID starting with 80d79e92fb0727aee8b0bef0dc1711db209ed843837da5f9eca77a11f76dcf48 not found: ID does not exist" Jul 02 17:28:41.733662 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.733639 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8"] Jul 02 17:28:41.740995 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:41.740977 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-cfcf23f1-587b85c4bf-b6vb8"] Jul 02 17:28:43.265518 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:43.265475 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e61530-aaff-4147-826d-80c4e0ae5a26" path="/var/lib/kubelet/pods/12e61530-aaff-4147-826d-80c4e0ae5a26/volumes" Jul 02 17:28:51.301023 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:28:51.300987 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm"] Jul 02 17:29:06.321045 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.321002 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" podUID="c86898f1-7cbf-49b7-b564-d97be1f3cdf1" containerName="main" containerID="cri-o://94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e" gracePeriod=45 Jul 02 17:29:06.569356 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.569328 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:29:06.633449 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.633422 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-model-cache\") pod \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " Jul 02 17:29:06.633618 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.633465 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-home\") pod \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " Jul 02 17:29:06.633618 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.633510 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnsvm\" (UniqueName: \"kubernetes.io/projected/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-kube-api-access-gnsvm\") pod \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " Jul 02 17:29:06.633618 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.633537 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-dshm\") pod \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " Jul 02 17:29:06.633818 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.633626 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tmp-dir\") pod \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " Jul 02 17:29:06.633818 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.633654 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tls-certs\") pod \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\" (UID: \"c86898f1-7cbf-49b7-b564-d97be1f3cdf1\") " Jul 02 17:29:06.633818 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.633742 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c86898f1-7cbf-49b7-b564-d97be1f3cdf1" (UID: "c86898f1-7cbf-49b7-b564-d97be1f3cdf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:29:06.633968 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.633829 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-home" (OuterVolumeSpecName: "home") pod "c86898f1-7cbf-49b7-b564-d97be1f3cdf1" (UID: "c86898f1-7cbf-49b7-b564-d97be1f3cdf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:29:06.634033 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.634000 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c86898f1-7cbf-49b7-b564-d97be1f3cdf1" (UID: "c86898f1-7cbf-49b7-b564-d97be1f3cdf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:29:06.634089 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.634051 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-model-cache\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:29:06.634089 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.634070 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-home\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:29:06.635827 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.635797 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-kube-api-access-gnsvm" (OuterVolumeSpecName: "kube-api-access-gnsvm") pod "c86898f1-7cbf-49b7-b564-d97be1f3cdf1" (UID: "c86898f1-7cbf-49b7-b564-d97be1f3cdf1"). InnerVolumeSpecName "kube-api-access-gnsvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:29:06.635827 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.635816 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-dshm" (OuterVolumeSpecName: "dshm") pod "c86898f1-7cbf-49b7-b564-d97be1f3cdf1" (UID: "c86898f1-7cbf-49b7-b564-d97be1f3cdf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:29:06.635960 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.635843 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c86898f1-7cbf-49b7-b564-d97be1f3cdf1" (UID: "c86898f1-7cbf-49b7-b564-d97be1f3cdf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:29:06.734935 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.734898 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tmp-dir\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:29:06.734935 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.734928 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-tls-certs\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:29:06.734935 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.734937 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnsvm\" (UniqueName: \"kubernetes.io/projected/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-kube-api-access-gnsvm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:29:06.735156 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.734949 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86898f1-7cbf-49b7-b564-d97be1f3cdf1-dshm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:29:06.830774 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.830706 2582 generic.go:356] "Generic (PLEG): container finished" podID="c86898f1-7cbf-49b7-b564-d97be1f3cdf1" containerID="94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e" exitCode=0 Jul 02 17:29:06.830774 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.830778 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" Jul 02 17:29:06.831019 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.830799 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" event={"ID":"c86898f1-7cbf-49b7-b564-d97be1f3cdf1","Type":"ContainerDied","Data":"94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e"} Jul 02 17:29:06.831019 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.830849 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm" event={"ID":"c86898f1-7cbf-49b7-b564-d97be1f3cdf1","Type":"ContainerDied","Data":"9106b4b9c7588d092beab1bbd9889761144a0cb183cc3ee0339f6b4be36757c9"} Jul 02 17:29:06.831019 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.830872 2582 scope.go:117] "RemoveContainer" containerID="94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e" Jul 02 17:29:06.845966 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.845945 2582 scope.go:117] "RemoveContainer" containerID="94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e" Jul 02 17:29:06.846246 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:29:06.846228 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e\": container with ID starting with 94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e not found: ID does not exist" containerID="94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e" Jul 02 17:29:06.846315 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.846270 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e"} err="failed to get container status \"94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e\": rpc error: code = NotFound desc = could not find container \"94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e\": container with ID starting with 94c009d73d21ed8d98fa915b0b7b175c5be1aaac1ee47f8ecc9e75665ac7289e not found: ID does not exist" Jul 02 17:29:06.854760 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.854718 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm"] Jul 02 17:29:06.856231 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:06.856213 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cfcf23f1/e2e-lifecycle-model-cfcf23f1-kserve-bdf8498c5-zdvgm"] Jul 02 17:29:07.266264 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:29:07.266234 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c86898f1-7cbf-49b7-b564-d97be1f3cdf1" path="/var/lib/kubelet/pods/c86898f1-7cbf-49b7-b564-d97be1f3cdf1/volumes" Jul 02 17:30:00.141719 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.141687 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29716890-rwj8r"] Jul 02 17:30:00.142163 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.142042 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e61530-aaff-4147-826d-80c4e0ae5a26" containerName="maas-api" Jul 02 17:30:00.142163 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.142055 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e61530-aaff-4147-826d-80c4e0ae5a26" containerName="maas-api" Jul 02 17:30:00.142163 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.142077 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c86898f1-7cbf-49b7-b564-d97be1f3cdf1" containerName="main" Jul 02 17:30:00.142163 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.142084 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c86898f1-7cbf-49b7-b564-d97be1f3cdf1" containerName="main" Jul 02 17:30:00.142163 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.142128 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e61530-aaff-4147-826d-80c4e0ae5a26" containerName="maas-api" Jul 02 17:30:00.142163 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.142136 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c86898f1-7cbf-49b7-b564-d97be1f3cdf1" containerName="main" Jul 02 17:30:00.149419 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.149391 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" Jul 02 17:30:00.151116 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.151089 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29716890-rwj8r"] Jul 02 17:30:00.151933 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.151907 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:30:00.296913 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.296877 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4qkm\" (UniqueName: \"kubernetes.io/projected/2b9d30e1-64d5-4579-90a9-fd0a9c459b14-kube-api-access-w4qkm\") pod \"maas-api-key-cleanup-29716890-rwj8r\" (UID: \"2b9d30e1-64d5-4579-90a9-fd0a9c459b14\") " pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" Jul 02 17:30:00.397484 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.397395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qkm\" (UniqueName: \"kubernetes.io/projected/2b9d30e1-64d5-4579-90a9-fd0a9c459b14-kube-api-access-w4qkm\") pod \"maas-api-key-cleanup-29716890-rwj8r\" (UID: \"2b9d30e1-64d5-4579-90a9-fd0a9c459b14\") " pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" Jul 02 17:30:00.406918 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.406894 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qkm\" (UniqueName: \"kubernetes.io/projected/2b9d30e1-64d5-4579-90a9-fd0a9c459b14-kube-api-access-w4qkm\") pod \"maas-api-key-cleanup-29716890-rwj8r\" (UID: \"2b9d30e1-64d5-4579-90a9-fd0a9c459b14\") " pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" Jul 02 17:30:00.460506 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.460479 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" Jul 02 17:30:00.587105 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:00.587076 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29716890-rwj8r"] Jul 02 17:30:01.076456 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:01.076362 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" event={"ID":"2b9d30e1-64d5-4579-90a9-fd0a9c459b14","Type":"ContainerStarted","Data":"9da0e530238f167107fac9e3378a5172ff071be83530055e0be0964ea5ebbdbf"} Jul 02 17:30:01.076456 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:01.076413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" event={"ID":"2b9d30e1-64d5-4579-90a9-fd0a9c459b14","Type":"ContainerStarted","Data":"3cda8d8d6f10040bce885df6bd83797524e4188bdd94d5d57fe73dde82670917"} Jul 02 17:30:01.091070 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:01.091019 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" podStartSLOduration=1.091004297 podStartE2EDuration="1.091004297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:30:01.090426608 +0000 UTC m=+1988.279743640" watchObservedRunningTime="2026-07-02 17:30:01.091004297 +0000 UTC m=+1988.280321328" Jul 02 17:30:02.083262 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:02.083220 2582 generic.go:356] "Generic (PLEG): container finished" podID="2b9d30e1-64d5-4579-90a9-fd0a9c459b14" containerID="9da0e530238f167107fac9e3378a5172ff071be83530055e0be0964ea5ebbdbf" exitCode=0 Jul 02 17:30:02.083647 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:02.083283 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" event={"ID":"2b9d30e1-64d5-4579-90a9-fd0a9c459b14","Type":"ContainerDied","Data":"9da0e530238f167107fac9e3378a5172ff071be83530055e0be0964ea5ebbdbf"} Jul 02 17:30:03.223953 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:03.223925 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" Jul 02 17:30:03.320905 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:03.320873 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4qkm\" (UniqueName: \"kubernetes.io/projected/2b9d30e1-64d5-4579-90a9-fd0a9c459b14-kube-api-access-w4qkm\") pod \"2b9d30e1-64d5-4579-90a9-fd0a9c459b14\" (UID: \"2b9d30e1-64d5-4579-90a9-fd0a9c459b14\") " Jul 02 17:30:03.322920 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:03.322887 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9d30e1-64d5-4579-90a9-fd0a9c459b14-kube-api-access-w4qkm" (OuterVolumeSpecName: "kube-api-access-w4qkm") pod "2b9d30e1-64d5-4579-90a9-fd0a9c459b14" (UID: "2b9d30e1-64d5-4579-90a9-fd0a9c459b14"). InnerVolumeSpecName "kube-api-access-w4qkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:30:03.422170 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:03.422120 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4qkm\" (UniqueName: \"kubernetes.io/projected/2b9d30e1-64d5-4579-90a9-fd0a9c459b14-kube-api-access-w4qkm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:30:04.094072 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:04.094030 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" event={"ID":"2b9d30e1-64d5-4579-90a9-fd0a9c459b14","Type":"ContainerDied","Data":"3cda8d8d6f10040bce885df6bd83797524e4188bdd94d5d57fe73dde82670917"} Jul 02 17:30:04.094251 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:04.094084 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cda8d8d6f10040bce885df6bd83797524e4188bdd94d5d57fe73dde82670917" Jul 02 17:30:04.094251 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:04.094043 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29716890-rwj8r" Jul 02 17:30:31.933222 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:31.933183 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf"] Jul 02 17:30:31.933760 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:31.933706 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9d30e1-64d5-4579-90a9-fd0a9c459b14" containerName="cleanup" Jul 02 17:30:31.933823 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:31.933760 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9d30e1-64d5-4579-90a9-fd0a9c459b14" containerName="cleanup" Jul 02 17:30:31.933900 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:31.933886 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9d30e1-64d5-4579-90a9-fd0a9c459b14" containerName="cleanup" Jul 02 17:30:31.937154 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:31.937132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:31.939584 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:31.939566 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:30:31.939967 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:31.939949 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-ec100e-serving-cert\"" Jul 02 17:30:31.944900 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:31.944880 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf"] Jul 02 17:30:32.069755 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.069694 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b574e1-4c19-426b-8268-d9b618269a85-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.069945 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.069834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfft\" (UniqueName: \"kubernetes.io/projected/a1b574e1-4c19-426b-8268-d9b618269a85-kube-api-access-rvfft\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.069945 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.069891 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.171317 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.171281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfft\" (UniqueName: \"kubernetes.io/projected/a1b574e1-4c19-426b-8268-d9b618269a85-kube-api-access-rvfft\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.171507 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.171333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.171507 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.171368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b574e1-4c19-426b-8268-d9b618269a85-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.171640 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:30:32.171504 2582 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-ec100e-serving-cert: secret "maas-api-e2e-inf-a-ec100e-serving-cert" not found Jul 02 17:30:32.171640 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:30:32.171587 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls podName:a1b574e1-4c19-426b-8268-d9b618269a85 nodeName:}" failed. No retries permitted until 2026-07-02 17:30:32.671566704 +0000 UTC m=+2019.860883719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls") pod "maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" (UID: "a1b574e1-4c19-426b-8268-d9b618269a85") : secret "maas-api-e2e-inf-a-ec100e-serving-cert" not found Jul 02 17:30:32.171968 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.171951 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b574e1-4c19-426b-8268-d9b618269a85-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.185837 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.185763 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfft\" (UniqueName: \"kubernetes.io/projected/a1b574e1-4c19-426b-8268-d9b618269a85-kube-api-access-rvfft\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.675800 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.675760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.678119 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.678100 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.851406 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.851369 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:32.984508 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.984483 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf"] Jul 02 17:30:32.989905 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:32.989874 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 17:30:33.221756 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:33.221659 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" event={"ID":"a1b574e1-4c19-426b-8268-d9b618269a85","Type":"ContainerStarted","Data":"0b819b052761ce51af2ce2ea66edf18ada95d4d41f89cb5d4dc4c76bd321ae25"} Jul 02 17:30:34.227987 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:34.227935 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" event={"ID":"a1b574e1-4c19-426b-8268-d9b618269a85","Type":"ContainerStarted","Data":"9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6"} Jul 02 17:30:34.228427 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:34.228033 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:34.243553 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:34.243514 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" podStartSLOduration=2.777349696 podStartE2EDuration="3.243500724s" totalImagesPullingTime="466.151028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:30:34.242629846 +0000 UTC m=+2021.431946881" watchObservedRunningTime="2026-07-02 17:30:34.243500724 +0000 UTC m=+2021.432817754" Jul 02 17:30:40.007155 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.007119 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk"] Jul 02 17:30:40.010803 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.010783 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.013647 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.013626 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-fcb6e8-serving-cert\"" Jul 02 17:30:40.027351 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.027319 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk"] Jul 02 17:30:40.145658 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.145617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.145839 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.145683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.145839 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.145810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxzc\" (UniqueName: \"kubernetes.io/projected/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-kube-api-access-qvxzc\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.238114 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.238087 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:30:40.246610 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.246575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxzc\" (UniqueName: \"kubernetes.io/projected/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-kube-api-access-qvxzc\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.246774 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.246616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.246842 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.246773 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.247407 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.247384 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.249014 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.248991 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.255813 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.255791 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxzc\" (UniqueName: \"kubernetes.io/projected/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-kube-api-access-qvxzc\") pod \"maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.320797 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.320705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:40.458498 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:40.458435 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk"] Jul 02 17:30:40.461151 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:30:40.461122 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b1cc66_ec2a_4e21_ad2f_87d7cf890e3d.slice/crio-34490cdf7d06854b6e490207d8fd7ea257b724016be0b880c8d55bdf16d78e9a WatchSource:0}: Error finding container 34490cdf7d06854b6e490207d8fd7ea257b724016be0b880c8d55bdf16d78e9a: Status 404 returned error can't find the container with id 34490cdf7d06854b6e490207d8fd7ea257b724016be0b880c8d55bdf16d78e9a Jul 02 17:30:41.265829 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:41.265798 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:41.266291 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:41.265845 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" event={"ID":"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d","Type":"ContainerStarted","Data":"65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8"} Jul 02 17:30:41.266291 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:41.265867 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" event={"ID":"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d","Type":"ContainerStarted","Data":"34490cdf7d06854b6e490207d8fd7ea257b724016be0b880c8d55bdf16d78e9a"} Jul 02 17:30:41.279960 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:41.279913 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" podStartSLOduration=1.925798667 podStartE2EDuration="2.279897682s" totalImagesPullingTime="354.099015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:30:41.277751252 +0000 UTC m=+2028.467068283" watchObservedRunningTime="2026-07-02 17:30:41.279897682 +0000 UTC m=+2028.469214713" Jul 02 17:30:47.272863 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:47.272836 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:30:49.778184 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.778148 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7"] Jul 02 17:30:49.781654 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.781635 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:49.785084 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.785064 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ec100e\"/\"default-dockercfg-9skxl\"" Jul 02 17:30:49.785188 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.785081 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ec100e\"/\"kube-root-ca.crt\"" Jul 02 17:30:49.785188 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.785139 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ec100e\"/\"test-model-ec100e-kserve-self-signed-certs\"" Jul 02 17:30:49.785188 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.785070 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ec100e\"/\"openshift-service-ca.crt\"" Jul 02 17:30:49.791901 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.791878 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7"] Jul 02 17:30:49.938623 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.938586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-home\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:49.938623 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.938621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-model-cache\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:49.938856 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.938654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tmp-dir\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:49.938856 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.938680 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzkct\" (UniqueName: \"kubernetes.io/projected/9b4c9ba2-1267-4ef8-828b-d4f03101f678-kube-api-access-lzkct\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:49.938856 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.938755 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tls-certs\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:49.938856 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:49.938795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-dshm\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.039855 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.039771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-home\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.039855 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.039808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-model-cache\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.040040 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.039861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tmp-dir\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.040040 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.039882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzkct\" (UniqueName: \"kubernetes.io/projected/9b4c9ba2-1267-4ef8-828b-d4f03101f678-kube-api-access-lzkct\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.040040 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.039941 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tls-certs\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.040040 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.039980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-dshm\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.040304 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.040282 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-model-cache\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.040375 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.040294 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tmp-dir\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.040375 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.040341 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-home\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.042372 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.042343 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-dshm\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.042662 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.042634 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tls-certs\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.047159 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.047140 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzkct\" (UniqueName: \"kubernetes.io/projected/9b4c9ba2-1267-4ef8-828b-d4f03101f678-kube-api-access-lzkct\") pod \"test-model-ec100e-kserve-c856f49f7-bzgh7\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.091032 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.091000 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:30:50.223667 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.223638 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7"] Jul 02 17:30:50.225875 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:30:50.225845 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b4c9ba2_1267_4ef8_828b_d4f03101f678.slice/crio-bd642c39c6ae827aed45674435acef0e9a84668dbb8e9a0f3a4cd0c8db04b26d WatchSource:0}: Error finding container bd642c39c6ae827aed45674435acef0e9a84668dbb8e9a0f3a4cd0c8db04b26d: Status 404 returned error can't find the container with id bd642c39c6ae827aed45674435acef0e9a84668dbb8e9a0f3a4cd0c8db04b26d Jul 02 17:30:50.304114 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.304040 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" event={"ID":"9b4c9ba2-1267-4ef8-828b-d4f03101f678","Type":"ContainerStarted","Data":"0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503"} Jul 02 17:30:50.304114 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.304083 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" event={"ID":"9b4c9ba2-1267-4ef8-828b-d4f03101f678","Type":"ContainerStarted","Data":"bd642c39c6ae827aed45674435acef0e9a84668dbb8e9a0f3a4cd0c8db04b26d"} Jul 02 17:30:50.321366 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:30:50.321320 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" podStartSLOduration=1.3213036040000001 podStartE2EDuration="1.321303604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:30:50.319565356 +0000 UTC m=+2037.508882390" watchObservedRunningTime="2026-07-02 17:30:50.321303604 +0000 UTC m=+2037.510620635" Jul 02 17:31:00.091177 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:00.091123 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:31:00.091745 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:00.091401 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:31:00.103869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:00.103846 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:31:00.359711 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:00.359631 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:31:08.572449 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.572413 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9"] Jul 02 17:31:08.575815 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.575800 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.578147 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.578128 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fcb6e8\"/\"openshift-service-ca.crt\"" Jul 02 17:31:08.578253 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.578148 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fcb6e8\"/\"kube-root-ca.crt\"" Jul 02 17:31:08.578981 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.578963 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fcb6e8\"/\"test-model-fcb6e8-kserve-self-signed-certs\"" Jul 02 17:31:08.579056 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.579006 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fcb6e8\"/\"default-dockercfg-9sq96\"" Jul 02 17:31:08.583551 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.583425 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9"] Jul 02 17:31:08.698572 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.698536 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tmp-dir\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.698572 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.698573 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrck\" (UniqueName: \"kubernetes.io/projected/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-kube-api-access-hkrck\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.699043 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.698600 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-home\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.699043 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.698642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-model-cache\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.699043 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.698771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tls-certs\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.699043 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.698804 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-dshm\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.799695 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.799654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-home\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.799695 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.799693 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-model-cache\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.799937 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.799784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tls-certs\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.799937 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.799814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-dshm\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.799937 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.799884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tmp-dir\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.799937 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.799908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrck\" (UniqueName: \"kubernetes.io/projected/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-kube-api-access-hkrck\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.800200 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.800177 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-home\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.800301 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.800279 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-model-cache\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.800359 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.800313 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tmp-dir\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.802160 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.802135 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-dshm\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.802390 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.802374 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tls-certs\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.807169 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.807150 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrck\" (UniqueName: \"kubernetes.io/projected/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-kube-api-access-hkrck\") pod \"test-model-fcb6e8-kserve-746655944f-gfrq9\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:08.886057 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:08.886027 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:09.016145 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:09.016120 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9"] Jul 02 17:31:09.017491 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:31:09.017456 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d24eefc_a88d_4981_8402_6e58b3dfb2c2.slice/crio-06da7b928e32415502eb7f189b97451fc39c20b877f8b25a7e25f28479ab430f WatchSource:0}: Error finding container 06da7b928e32415502eb7f189b97451fc39c20b877f8b25a7e25f28479ab430f: Status 404 returned error can't find the container with id 06da7b928e32415502eb7f189b97451fc39c20b877f8b25a7e25f28479ab430f Jul 02 17:31:09.393009 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:09.392976 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" event={"ID":"3d24eefc-a88d-4981-8402-6e58b3dfb2c2","Type":"ContainerStarted","Data":"1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a"} Jul 02 17:31:09.393193 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:09.393016 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" event={"ID":"3d24eefc-a88d-4981-8402-6e58b3dfb2c2","Type":"ContainerStarted","Data":"06da7b928e32415502eb7f189b97451fc39c20b877f8b25a7e25f28479ab430f"} Jul 02 17:31:09.411004 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:09.410952 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" podStartSLOduration=1.410935629 podStartE2EDuration="1.410935629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:31:09.408468132 +0000 UTC m=+2056.597785185" watchObservedRunningTime="2026-07-02 17:31:09.410935629 +0000 UTC m=+2056.600252661" Jul 02 17:31:18.887216 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:18.887127 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:18.887216 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:18.887190 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:18.901380 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:18.901348 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:19.459100 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:19.459070 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:27.572020 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:27.571986 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7"] Jul 02 17:31:28.522458 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:28.522425 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9"] Jul 02 17:31:29.360765 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.360714 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf"] Jul 02 17:31:29.361197 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.360987 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" podUID="a1b574e1-4c19-426b-8268-d9b618269a85" containerName="maas-api" containerID="cri-o://9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6" gracePeriod=30 Jul 02 17:31:29.810492 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.810466 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:31:29.893698 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.893676 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b574e1-4c19-426b-8268-d9b618269a85-openshift-service-ca\") pod \"a1b574e1-4c19-426b-8268-d9b618269a85\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " Jul 02 17:31:29.893866 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.893761 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvfft\" (UniqueName: \"kubernetes.io/projected/a1b574e1-4c19-426b-8268-d9b618269a85-kube-api-access-rvfft\") pod \"a1b574e1-4c19-426b-8268-d9b618269a85\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " Jul 02 17:31:29.893944 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.893923 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls\") pod \"a1b574e1-4c19-426b-8268-d9b618269a85\" (UID: \"a1b574e1-4c19-426b-8268-d9b618269a85\") " Jul 02 17:31:29.894083 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.894056 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b574e1-4c19-426b-8268-d9b618269a85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1b574e1-4c19-426b-8268-d9b618269a85" (UID: "a1b574e1-4c19-426b-8268-d9b618269a85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:31:29.894254 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.894237 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b574e1-4c19-426b-8268-d9b618269a85-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:29.895766 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.895739 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b574e1-4c19-426b-8268-d9b618269a85-kube-api-access-rvfft" (OuterVolumeSpecName: "kube-api-access-rvfft") pod "a1b574e1-4c19-426b-8268-d9b618269a85" (UID: "a1b574e1-4c19-426b-8268-d9b618269a85"). InnerVolumeSpecName "kube-api-access-rvfft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:31:29.895888 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.895832 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1b574e1-4c19-426b-8268-d9b618269a85" (UID: "a1b574e1-4c19-426b-8268-d9b618269a85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:31:29.995444 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.995404 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfft\" (UniqueName: \"kubernetes.io/projected/a1b574e1-4c19-426b-8268-d9b618269a85-kube-api-access-rvfft\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:29.995444 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:29.995438 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b574e1-4c19-426b-8268-d9b618269a85-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:30.499463 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.499423 2582 generic.go:356] "Generic (PLEG): container finished" podID="a1b574e1-4c19-426b-8268-d9b618269a85" containerID="9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6" exitCode=0 Jul 02 17:31:30.499923 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.499480 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" Jul 02 17:31:30.499923 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.499515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" event={"ID":"a1b574e1-4c19-426b-8268-d9b618269a85","Type":"ContainerDied","Data":"9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6"} Jul 02 17:31:30.499923 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.499572 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf" event={"ID":"a1b574e1-4c19-426b-8268-d9b618269a85","Type":"ContainerDied","Data":"0b819b052761ce51af2ce2ea66edf18ada95d4d41f89cb5d4dc4c76bd321ae25"} Jul 02 17:31:30.499923 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.499595 2582 scope.go:117] "RemoveContainer" containerID="9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6" Jul 02 17:31:30.511583 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.511567 2582 scope.go:117] "RemoveContainer" containerID="9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6" Jul 02 17:31:30.511887 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:31:30.511866 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6\": container with ID starting with 9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6 not found: ID does not exist" containerID="9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6" Jul 02 17:31:30.511966 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.511899 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6"} err="failed to get container status \"9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6\": rpc error: code = NotFound desc = could not find container \"9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6\": container with ID starting with 9c34ad63fb6756d82d4a60d5561952061fb89e1680cfae0dbef38493aca13ef6 not found: ID does not exist" Jul 02 17:31:30.525436 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.525400 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf"] Jul 02 17:31:30.526774 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:30.526753 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-ec100e-589c6b69bc-fr7tf"] Jul 02 17:31:31.266183 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:31.266154 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b574e1-4c19-426b-8268-d9b618269a85" path="/var/lib/kubelet/pods/a1b574e1-4c19-426b-8268-d9b618269a85/volumes" Jul 02 17:31:42.592533 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.592477 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" podUID="9b4c9ba2-1267-4ef8-828b-d4f03101f678" containerName="main" containerID="cri-o://0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503" gracePeriod=45 Jul 02 17:31:42.841462 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.841437 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:31:42.903997 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.903971 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-model-cache\") pod \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " Jul 02 17:31:42.904137 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904015 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-dshm\") pod \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " Jul 02 17:31:42.904137 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904087 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzkct\" (UniqueName: \"kubernetes.io/projected/9b4c9ba2-1267-4ef8-828b-d4f03101f678-kube-api-access-lzkct\") pod \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " Jul 02 17:31:42.904137 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904114 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-home\") pod \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " Jul 02 17:31:42.904137 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904130 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tmp-dir\") pod \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " Jul 02 17:31:42.904354 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904147 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tls-certs\") pod \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\" (UID: \"9b4c9ba2-1267-4ef8-828b-d4f03101f678\") " Jul 02 17:31:42.904354 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904245 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b4c9ba2-1267-4ef8-828b-d4f03101f678" (UID: "9b4c9ba2-1267-4ef8-828b-d4f03101f678"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:31:42.904459 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904386 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-home" (OuterVolumeSpecName: "home") pod "9b4c9ba2-1267-4ef8-828b-d4f03101f678" (UID: "9b4c9ba2-1267-4ef8-828b-d4f03101f678"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:31:42.904459 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904405 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b4c9ba2-1267-4ef8-828b-d4f03101f678" (UID: "9b4c9ba2-1267-4ef8-828b-d4f03101f678"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:31:42.904541 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.904454 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-model-cache\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:42.906235 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.906208 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-dshm" (OuterVolumeSpecName: "dshm") pod "9b4c9ba2-1267-4ef8-828b-d4f03101f678" (UID: "9b4c9ba2-1267-4ef8-828b-d4f03101f678"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:31:42.906335 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.906240 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b4c9ba2-1267-4ef8-828b-d4f03101f678" (UID: "9b4c9ba2-1267-4ef8-828b-d4f03101f678"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:31:42.906377 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:42.906331 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4c9ba2-1267-4ef8-828b-d4f03101f678-kube-api-access-lzkct" (OuterVolumeSpecName: "kube-api-access-lzkct") pod "9b4c9ba2-1267-4ef8-828b-d4f03101f678" (UID: "9b4c9ba2-1267-4ef8-828b-d4f03101f678"). InnerVolumeSpecName "kube-api-access-lzkct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:31:43.005255 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.005226 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-dshm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.005255 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.005250 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzkct\" (UniqueName: \"kubernetes.io/projected/9b4c9ba2-1267-4ef8-828b-d4f03101f678-kube-api-access-lzkct\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.005255 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.005260 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-home\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.005472 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.005268 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tmp-dir\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.005472 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.005276 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4c9ba2-1267-4ef8-828b-d4f03101f678-tls-certs\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.549340 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.549252 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" podUID="3d24eefc-a88d-4981-8402-6e58b3dfb2c2" containerName="main" containerID="cri-o://1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a" gracePeriod=45 Jul 02 17:31:43.559940 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.559917 2582 generic.go:356] "Generic (PLEG): container finished" podID="9b4c9ba2-1267-4ef8-828b-d4f03101f678" containerID="0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503" exitCode=0 Jul 02 17:31:43.560031 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.559968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" event={"ID":"9b4c9ba2-1267-4ef8-828b-d4f03101f678","Type":"ContainerDied","Data":"0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503"} Jul 02 17:31:43.560031 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.559974 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" Jul 02 17:31:43.560031 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.559997 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7" event={"ID":"9b4c9ba2-1267-4ef8-828b-d4f03101f678","Type":"ContainerDied","Data":"bd642c39c6ae827aed45674435acef0e9a84668dbb8e9a0f3a4cd0c8db04b26d"} Jul 02 17:31:43.560031 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.560014 2582 scope.go:117] "RemoveContainer" containerID="0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503" Jul 02 17:31:43.627780 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.627751 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7"] Jul 02 17:31:43.631498 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.631474 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ec100e/test-model-ec100e-kserve-c856f49f7-bzgh7"] Jul 02 17:31:43.659831 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.659660 2582 scope.go:117] "RemoveContainer" containerID="0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503" Jul 02 17:31:43.659939 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:31:43.659921 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503\": container with ID starting with 0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503 not found: ID does not exist" containerID="0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503" Jul 02 17:31:43.659984 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.659948 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503"} err="failed to get container status \"0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503\": rpc error: code = NotFound desc = could not find container \"0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503\": container with ID starting with 0ccbbec6f70b8db0b6182a77a67938d4a4e23d451577b995c56790eef0656503 not found: ID does not exist" Jul 02 17:31:43.803516 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.803463 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:43.809695 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.809668 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tmp-dir\") pod \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " Jul 02 17:31:43.809869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.809699 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-model-cache\") pod \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " Jul 02 17:31:43.809869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.809761 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkrck\" (UniqueName: \"kubernetes.io/projected/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-kube-api-access-hkrck\") pod \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " Jul 02 17:31:43.809869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.809786 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tls-certs\") pod \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " Jul 02 17:31:43.809869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.809805 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-dshm\") pod \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " Jul 02 17:31:43.809869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.809828 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-home\") pod \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\" (UID: \"3d24eefc-a88d-4981-8402-6e58b3dfb2c2\") " Jul 02 17:31:43.810140 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.809949 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d24eefc-a88d-4981-8402-6e58b3dfb2c2" (UID: "3d24eefc-a88d-4981-8402-6e58b3dfb2c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:31:43.810140 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.810096 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d24eefc-a88d-4981-8402-6e58b3dfb2c2" (UID: "3d24eefc-a88d-4981-8402-6e58b3dfb2c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:31:43.810244 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.810139 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tmp-dir\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.810244 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.810173 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-home" (OuterVolumeSpecName: "home") pod "3d24eefc-a88d-4981-8402-6e58b3dfb2c2" (UID: "3d24eefc-a88d-4981-8402-6e58b3dfb2c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:31:43.811991 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.811969 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-dshm" (OuterVolumeSpecName: "dshm") pod "3d24eefc-a88d-4981-8402-6e58b3dfb2c2" (UID: "3d24eefc-a88d-4981-8402-6e58b3dfb2c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:31:43.812066 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.812031 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d24eefc-a88d-4981-8402-6e58b3dfb2c2" (UID: "3d24eefc-a88d-4981-8402-6e58b3dfb2c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:31:43.812141 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.812125 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-kube-api-access-hkrck" (OuterVolumeSpecName: "kube-api-access-hkrck") pod "3d24eefc-a88d-4981-8402-6e58b3dfb2c2" (UID: "3d24eefc-a88d-4981-8402-6e58b3dfb2c2"). InnerVolumeSpecName "kube-api-access-hkrck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:31:43.910859 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.910826 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-model-cache\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.910859 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.910855 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrck\" (UniqueName: \"kubernetes.io/projected/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-kube-api-access-hkrck\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.910859 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.910865 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-tls-certs\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.911098 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.910874 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-dshm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:43.911098 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:43.910881 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24eefc-a88d-4981-8402-6e58b3dfb2c2-home\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:31:44.565171 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.565139 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d24eefc-a88d-4981-8402-6e58b3dfb2c2" containerID="1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a" exitCode=0 Jul 02 17:31:44.565352 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.565183 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" Jul 02 17:31:44.565352 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.565217 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" event={"ID":"3d24eefc-a88d-4981-8402-6e58b3dfb2c2","Type":"ContainerDied","Data":"1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a"} Jul 02 17:31:44.565352 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.565261 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9" event={"ID":"3d24eefc-a88d-4981-8402-6e58b3dfb2c2","Type":"ContainerDied","Data":"06da7b928e32415502eb7f189b97451fc39c20b877f8b25a7e25f28479ab430f"} Jul 02 17:31:44.565352 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.565282 2582 scope.go:117] "RemoveContainer" containerID="1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a" Jul 02 17:31:44.573644 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.573619 2582 scope.go:117] "RemoveContainer" containerID="1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a" Jul 02 17:31:44.573904 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:31:44.573884 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a\": container with ID starting with 1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a not found: ID does not exist" containerID="1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a" Jul 02 17:31:44.573954 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.573912 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a"} err="failed to get container status \"1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a\": rpc error: code = NotFound desc = could not find container \"1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a\": container with ID starting with 1d3920b724b577ebf5155a005a1b8816d0cfddeec73b98253051e1a197e5dd7a not found: ID does not exist" Jul 02 17:31:44.586043 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.586022 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9"] Jul 02 17:31:44.587604 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:44.587581 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fcb6e8/test-model-fcb6e8-kserve-746655944f-gfrq9"] Jul 02 17:31:45.265367 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:45.265336 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d24eefc-a88d-4981-8402-6e58b3dfb2c2" path="/var/lib/kubelet/pods/3d24eefc-a88d-4981-8402-6e58b3dfb2c2/volumes" Jul 02 17:31:45.265766 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:45.265716 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b4c9ba2-1267-4ef8-828b-d4f03101f678" path="/var/lib/kubelet/pods/9b4c9ba2-1267-4ef8-828b-d4f03101f678/volumes" Jul 02 17:31:53.356880 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:53.356854 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:31:53.366714 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:31:53.366692 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:32:12.404635 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.404601 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk"] Jul 02 17:32:12.405090 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.404819 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" podUID="31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" containerName="maas-api" containerID="cri-o://65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8" gracePeriod=30 Jul 02 17:32:12.843907 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.843884 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:32:12.954779 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.954673 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-maas-api-tls\") pod \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " Jul 02 17:32:12.954779 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.954716 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-openshift-service-ca\") pod \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " Jul 02 17:32:12.954779 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.954768 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvxzc\" (UniqueName: \"kubernetes.io/projected/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-kube-api-access-qvxzc\") pod \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\" (UID: \"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d\") " Jul 02 17:32:12.955133 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.955101 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" (UID: "31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:32:12.956837 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.956799 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" (UID: "31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:32:12.956931 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:12.956906 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-kube-api-access-qvxzc" (OuterVolumeSpecName: "kube-api-access-qvxzc") pod "31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" (UID: "31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d"). InnerVolumeSpecName "kube-api-access-qvxzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:32:13.055964 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.055934 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:13.055964 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.055959 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:13.055964 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.055971 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvxzc\" (UniqueName: \"kubernetes.io/projected/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d-kube-api-access-qvxzc\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:13.705111 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.705077 2582 generic.go:356] "Generic (PLEG): container finished" podID="31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" containerID="65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8" exitCode=0 Jul 02 17:32:13.705502 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.705125 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" Jul 02 17:32:13.705502 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.705127 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" event={"ID":"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d","Type":"ContainerDied","Data":"65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8"} Jul 02 17:32:13.705502 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.705246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk" event={"ID":"31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d","Type":"ContainerDied","Data":"34490cdf7d06854b6e490207d8fd7ea257b724016be0b880c8d55bdf16d78e9a"} Jul 02 17:32:13.705502 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.705288 2582 scope.go:117] "RemoveContainer" containerID="65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8" Jul 02 17:32:13.713433 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.713418 2582 scope.go:117] "RemoveContainer" containerID="65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8" Jul 02 17:32:13.713684 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:32:13.713666 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8\": container with ID starting with 65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8 not found: ID does not exist" containerID="65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8" Jul 02 17:32:13.713758 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.713690 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8"} err="failed to get container status \"65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8\": rpc error: code = NotFound desc = could not find container \"65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8\": container with ID starting with 65c32125f8e428eec7733f1ac88a1fc0d28210ad6f7d5edde8dab5c3c3e2b7e8 not found: ID does not exist" Jul 02 17:32:13.720461 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.720441 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk"] Jul 02 17:32:13.723832 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:13.723812 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-fcb6e8-6f8475cc78-9xrvk"] Jul 02 17:32:15.265132 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:15.265098 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" path="/var/lib/kubelet/pods/31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d/volumes" Jul 02 17:32:22.506321 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506284 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv"] Jul 02 17:32:22.506696 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506614 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d24eefc-a88d-4981-8402-6e58b3dfb2c2" containerName="main" Jul 02 17:32:22.506696 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506626 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d24eefc-a88d-4981-8402-6e58b3dfb2c2" containerName="main" Jul 02 17:32:22.506696 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506639 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4c9ba2-1267-4ef8-828b-d4f03101f678" containerName="main" Jul 02 17:32:22.506696 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506644 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4c9ba2-1267-4ef8-828b-d4f03101f678" containerName="main" Jul 02 17:32:22.506696 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506655 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" containerName="maas-api" Jul 02 17:32:22.506696 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506661 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" containerName="maas-api" Jul 02 17:32:22.506696 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506669 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b574e1-4c19-426b-8268-d9b618269a85" containerName="maas-api" Jul 02 17:32:22.506696 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506674 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b574e1-4c19-426b-8268-d9b618269a85" containerName="maas-api" Jul 02 17:32:22.506992 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506741 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b574e1-4c19-426b-8268-d9b618269a85" containerName="maas-api" Jul 02 17:32:22.506992 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506756 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="31b1cc66-ec2a-4e21-ad2f-87d7cf890e3d" containerName="maas-api" Jul 02 17:32:22.506992 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506762 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d24eefc-a88d-4981-8402-6e58b3dfb2c2" containerName="main" Jul 02 17:32:22.506992 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.506768 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4c9ba2-1267-4ef8-828b-d4f03101f678" containerName="main" Jul 02 17:32:22.509403 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.509386 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.512804 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.512784 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-19fdf3-serving-cert\"" Jul 02 17:32:22.512889 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.512870 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:32:22.529991 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.529970 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv"] Jul 02 17:32:22.640948 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.640915 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d98f363-1bec-4735-b9d3-d1379d3c507d-maas-api-tls\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.641132 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.640976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d98f363-1bec-4735-b9d3-d1379d3c507d-openshift-service-ca\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.641132 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.641040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmk7x\" (UniqueName: \"kubernetes.io/projected/4d98f363-1bec-4735-b9d3-d1379d3c507d-kube-api-access-rmk7x\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.742098 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.742059 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d98f363-1bec-4735-b9d3-d1379d3c507d-openshift-service-ca\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.742299 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.742132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmk7x\" (UniqueName: \"kubernetes.io/projected/4d98f363-1bec-4735-b9d3-d1379d3c507d-kube-api-access-rmk7x\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.742299 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.742177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d98f363-1bec-4735-b9d3-d1379d3c507d-maas-api-tls\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.742952 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.742928 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d98f363-1bec-4735-b9d3-d1379d3c507d-openshift-service-ca\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.744663 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.744638 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d98f363-1bec-4735-b9d3-d1379d3c507d-maas-api-tls\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.767334 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.767254 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmk7x\" (UniqueName: \"kubernetes.io/projected/4d98f363-1bec-4735-b9d3-d1379d3c507d-kube-api-access-rmk7x\") pod \"maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.819880 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.819844 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:22.974567 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:22.974476 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv"] Jul 02 17:32:22.976907 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:32:22.976878 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d98f363_1bec_4735_b9d3_d1379d3c507d.slice/crio-c73a8a4e13350040d5894af772b1b40458dd1421807610e4a3714d47b8d4b670 WatchSource:0}: Error finding container c73a8a4e13350040d5894af772b1b40458dd1421807610e4a3714d47b8d4b670: Status 404 returned error can't find the container with id c73a8a4e13350040d5894af772b1b40458dd1421807610e4a3714d47b8d4b670 Jul 02 17:32:23.752918 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:23.752829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" event={"ID":"4d98f363-1bec-4735-b9d3-d1379d3c507d","Type":"ContainerStarted","Data":"01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53"} Jul 02 17:32:23.752918 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:23.752910 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" event={"ID":"4d98f363-1bec-4735-b9d3-d1379d3c507d","Type":"ContainerStarted","Data":"c73a8a4e13350040d5894af772b1b40458dd1421807610e4a3714d47b8d4b670"} Jul 02 17:32:23.753328 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:23.752962 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:23.781442 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:23.781390 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" podStartSLOduration=1.292442192 podStartE2EDuration="1.781374086s" totalImagesPullingTime="488.931894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:32:23.779651994 +0000 UTC m=+2130.968969029" watchObservedRunningTime="2026-07-02 17:32:23.781374086 +0000 UTC m=+2130.970691116" Jul 02 17:32:29.762426 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:29.762391 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:30.447497 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.447462 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9"] Jul 02 17:32:30.450182 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.450147 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.453369 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.453347 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-5a4c4c-serving-cert\"" Jul 02 17:32:30.464362 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.464334 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9"] Jul 02 17:32:30.517607 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.517572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhgd\" (UniqueName: \"kubernetes.io/projected/8a113976-54af-415c-9e6b-0dbec48e4012-kube-api-access-vvhgd\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.517842 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.517618 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a113976-54af-415c-9e6b-0dbec48e4012-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.517842 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.517642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a113976-54af-415c-9e6b-0dbec48e4012-maas-api-tls\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.618664 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.618626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhgd\" (UniqueName: \"kubernetes.io/projected/8a113976-54af-415c-9e6b-0dbec48e4012-kube-api-access-vvhgd\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.618881 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.618687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a113976-54af-415c-9e6b-0dbec48e4012-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.618881 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.618775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a113976-54af-415c-9e6b-0dbec48e4012-maas-api-tls\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.619486 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.619462 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a113976-54af-415c-9e6b-0dbec48e4012-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.621260 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.621238 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a113976-54af-415c-9e6b-0dbec48e4012-maas-api-tls\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.629246 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.629221 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhgd\" (UniqueName: \"kubernetes.io/projected/8a113976-54af-415c-9e6b-0dbec48e4012-kube-api-access-vvhgd\") pod \"maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.763519 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.763436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:30.896283 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:30.896253 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9"] Jul 02 17:32:30.898319 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:32:30.898293 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a113976_54af_415c_9e6b_0dbec48e4012.slice/crio-234be87977aa453a2aa2e550aa6e7dc6ce1a585a435c857b9ed8a757336ba738 WatchSource:0}: Error finding container 234be87977aa453a2aa2e550aa6e7dc6ce1a585a435c857b9ed8a757336ba738: Status 404 returned error can't find the container with id 234be87977aa453a2aa2e550aa6e7dc6ce1a585a435c857b9ed8a757336ba738 Jul 02 17:32:31.800044 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:31.800010 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" event={"ID":"8a113976-54af-415c-9e6b-0dbec48e4012","Type":"ContainerStarted","Data":"501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48"} Jul 02 17:32:31.800503 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:31.800066 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" event={"ID":"8a113976-54af-415c-9e6b-0dbec48e4012","Type":"ContainerStarted","Data":"234be87977aa453a2aa2e550aa6e7dc6ce1a585a435c857b9ed8a757336ba738"} Jul 02 17:32:31.800503 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:31.800216 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:31.817046 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:31.816996 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" podStartSLOduration=1.351452713 podStartE2EDuration="1.816979898s" totalImagesPullingTime="465.527185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:32:31.815620026 +0000 UTC m=+2139.004937058" watchObservedRunningTime="2026-07-02 17:32:31.816979898 +0000 UTC m=+2139.006296952" Jul 02 17:32:37.810249 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:37.810219 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:43.865623 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:43.865587 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv"] Jul 02 17:32:43.866086 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:43.865811 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" podUID="4d98f363-1bec-4735-b9d3-d1379d3c507d" containerName="maas-api" containerID="cri-o://01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53" gracePeriod=30 Jul 02 17:32:44.107027 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.106999 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:44.121315 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.121259 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d98f363-1bec-4735-b9d3-d1379d3c507d-maas-api-tls\") pod \"4d98f363-1bec-4735-b9d3-d1379d3c507d\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " Jul 02 17:32:44.121431 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.121327 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d98f363-1bec-4735-b9d3-d1379d3c507d-openshift-service-ca\") pod \"4d98f363-1bec-4735-b9d3-d1379d3c507d\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " Jul 02 17:32:44.121431 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.121347 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmk7x\" (UniqueName: \"kubernetes.io/projected/4d98f363-1bec-4735-b9d3-d1379d3c507d-kube-api-access-rmk7x\") pod \"4d98f363-1bec-4735-b9d3-d1379d3c507d\" (UID: \"4d98f363-1bec-4735-b9d3-d1379d3c507d\") " Jul 02 17:32:44.121744 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.121702 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d98f363-1bec-4735-b9d3-d1379d3c507d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d98f363-1bec-4735-b9d3-d1379d3c507d" (UID: "4d98f363-1bec-4735-b9d3-d1379d3c507d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:32:44.123932 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.123905 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d98f363-1bec-4735-b9d3-d1379d3c507d-kube-api-access-rmk7x" (OuterVolumeSpecName: "kube-api-access-rmk7x") pod "4d98f363-1bec-4735-b9d3-d1379d3c507d" (UID: "4d98f363-1bec-4735-b9d3-d1379d3c507d"). InnerVolumeSpecName "kube-api-access-rmk7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:32:44.124025 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.123977 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d98f363-1bec-4735-b9d3-d1379d3c507d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d98f363-1bec-4735-b9d3-d1379d3c507d" (UID: "4d98f363-1bec-4735-b9d3-d1379d3c507d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:32:44.222078 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.222047 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d98f363-1bec-4735-b9d3-d1379d3c507d-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:44.222078 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.222075 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmk7x\" (UniqueName: \"kubernetes.io/projected/4d98f363-1bec-4735-b9d3-d1379d3c507d-kube-api-access-rmk7x\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:44.222078 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.222085 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d98f363-1bec-4735-b9d3-d1379d3c507d-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:44.859762 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.859708 2582 generic.go:356] "Generic (PLEG): container finished" podID="4d98f363-1bec-4735-b9d3-d1379d3c507d" containerID="01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53" exitCode=0 Jul 02 17:32:44.859920 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.859779 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" Jul 02 17:32:44.859920 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.859803 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" event={"ID":"4d98f363-1bec-4735-b9d3-d1379d3c507d","Type":"ContainerDied","Data":"01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53"} Jul 02 17:32:44.859920 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.859858 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv" event={"ID":"4d98f363-1bec-4735-b9d3-d1379d3c507d","Type":"ContainerDied","Data":"c73a8a4e13350040d5894af772b1b40458dd1421807610e4a3714d47b8d4b670"} Jul 02 17:32:44.859920 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.859886 2582 scope.go:117] "RemoveContainer" containerID="01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53" Jul 02 17:32:44.868498 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.868349 2582 scope.go:117] "RemoveContainer" containerID="01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53" Jul 02 17:32:44.868756 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:32:44.868556 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53\": container with ID starting with 01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53 not found: ID does not exist" containerID="01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53" Jul 02 17:32:44.868756 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.868579 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53"} err="failed to get container status \"01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53\": rpc error: code = NotFound desc = could not find container \"01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53\": container with ID starting with 01feafa33dbd4e3e88013222dc326aee134eb7201c74151ff496ce5019101d53 not found: ID does not exist" Jul 02 17:32:44.880036 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.880014 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv"] Jul 02 17:32:44.884210 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:44.884191 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-19fdf3-5b98bff7ff-gpqhv"] Jul 02 17:32:45.265303 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:45.265269 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d98f363-1bec-4735-b9d3-d1379d3c507d" path="/var/lib/kubelet/pods/4d98f363-1bec-4735-b9d3-d1379d3c507d/volumes" Jul 02 17:32:50.452824 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.452781 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9"] Jul 02 17:32:50.453279 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.453057 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" podUID="8a113976-54af-415c-9e6b-0dbec48e4012" containerName="maas-api" containerID="cri-o://501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48" gracePeriod=30 Jul 02 17:32:50.697358 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.697335 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:50.773257 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.773170 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a113976-54af-415c-9e6b-0dbec48e4012-openshift-service-ca\") pod \"8a113976-54af-415c-9e6b-0dbec48e4012\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " Jul 02 17:32:50.773257 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.773218 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvhgd\" (UniqueName: \"kubernetes.io/projected/8a113976-54af-415c-9e6b-0dbec48e4012-kube-api-access-vvhgd\") pod \"8a113976-54af-415c-9e6b-0dbec48e4012\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " Jul 02 17:32:50.773486 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.773268 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a113976-54af-415c-9e6b-0dbec48e4012-maas-api-tls\") pod \"8a113976-54af-415c-9e6b-0dbec48e4012\" (UID: \"8a113976-54af-415c-9e6b-0dbec48e4012\") " Jul 02 17:32:50.773550 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.773508 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a113976-54af-415c-9e6b-0dbec48e4012-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a113976-54af-415c-9e6b-0dbec48e4012" (UID: "8a113976-54af-415c-9e6b-0dbec48e4012"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:32:50.775367 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.775335 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a113976-54af-415c-9e6b-0dbec48e4012-kube-api-access-vvhgd" (OuterVolumeSpecName: "kube-api-access-vvhgd") pod "8a113976-54af-415c-9e6b-0dbec48e4012" (UID: "8a113976-54af-415c-9e6b-0dbec48e4012"). InnerVolumeSpecName "kube-api-access-vvhgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:32:50.775450 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.775375 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a113976-54af-415c-9e6b-0dbec48e4012-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a113976-54af-415c-9e6b-0dbec48e4012" (UID: "8a113976-54af-415c-9e6b-0dbec48e4012"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:32:50.874641 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.874606 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a113976-54af-415c-9e6b-0dbec48e4012-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:50.874641 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.874635 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvhgd\" (UniqueName: \"kubernetes.io/projected/8a113976-54af-415c-9e6b-0dbec48e4012-kube-api-access-vvhgd\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:50.874641 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.874647 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a113976-54af-415c-9e6b-0dbec48e4012-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:32:50.887404 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.887378 2582 generic.go:356] "Generic (PLEG): container finished" podID="8a113976-54af-415c-9e6b-0dbec48e4012" containerID="501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48" exitCode=0 Jul 02 17:32:50.887540 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.887417 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" event={"ID":"8a113976-54af-415c-9e6b-0dbec48e4012","Type":"ContainerDied","Data":"501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48"} Jul 02 17:32:50.887540 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.887424 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" Jul 02 17:32:50.887540 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.887470 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9" event={"ID":"8a113976-54af-415c-9e6b-0dbec48e4012","Type":"ContainerDied","Data":"234be87977aa453a2aa2e550aa6e7dc6ce1a585a435c857b9ed8a757336ba738"} Jul 02 17:32:50.887540 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.887500 2582 scope.go:117] "RemoveContainer" containerID="501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48" Jul 02 17:32:50.897221 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.897207 2582 scope.go:117] "RemoveContainer" containerID="501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48" Jul 02 17:32:50.897459 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:32:50.897439 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48\": container with ID starting with 501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48 not found: ID does not exist" containerID="501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48" Jul 02 17:32:50.897521 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.897467 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48"} err="failed to get container status \"501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48\": rpc error: code = NotFound desc = could not find container \"501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48\": container with ID starting with 501428278d1553ce0229b7d0172ce3b590a1952b8d8e931e8e1992b8b7f94a48 not found: ID does not exist" Jul 02 17:32:50.907785 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.907767 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9"] Jul 02 17:32:50.911573 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:50.911551 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a4c4c-6c7fd7df57-jx4n9"] Jul 02 17:32:51.265498 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:32:51.265461 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a113976-54af-415c-9e6b-0dbec48e4012" path="/var/lib/kubelet/pods/8a113976-54af-415c-9e6b-0dbec48e4012/volumes" Jul 02 17:33:00.613278 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.613242 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2"] Jul 02 17:33:00.613895 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.613718 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d98f363-1bec-4735-b9d3-d1379d3c507d" containerName="maas-api" Jul 02 17:33:00.613895 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.613753 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d98f363-1bec-4735-b9d3-d1379d3c507d" containerName="maas-api" Jul 02 17:33:00.613895 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.613778 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a113976-54af-415c-9e6b-0dbec48e4012" containerName="maas-api" Jul 02 17:33:00.613895 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.613788 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a113976-54af-415c-9e6b-0dbec48e4012" containerName="maas-api" Jul 02 17:33:00.613895 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.613893 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a113976-54af-415c-9e6b-0dbec48e4012" containerName="maas-api" Jul 02 17:33:00.614169 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.613905 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d98f363-1bec-4735-b9d3-d1379d3c507d" containerName="maas-api" Jul 02 17:33:00.616567 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.616547 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:00.619891 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.619870 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-73dc61-serving-cert\"" Jul 02 17:33:00.620563 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.620546 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6kj8x\"" Jul 02 17:33:00.628669 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.628648 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2"] Jul 02 17:33:00.660258 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.660229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:00.660422 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.660267 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgpq\" (UniqueName: \"kubernetes.io/projected/3c568fe3-f52b-46e4-97ed-d385287f6a7c-kube-api-access-5lgpq\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:00.660422 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.660295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c568fe3-f52b-46e4-97ed-d385287f6a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:00.761628 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.761588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:00.761628 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.761622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgpq\" (UniqueName: \"kubernetes.io/projected/3c568fe3-f52b-46e4-97ed-d385287f6a7c-kube-api-access-5lgpq\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:00.761903 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.761653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c568fe3-f52b-46e4-97ed-d385287f6a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:00.761903 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:33:00.761765 2582 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-73dc61-serving-cert: secret "maas-api-e2e-shared-a-73dc61-serving-cert" not found Jul 02 17:33:00.761903 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:33:00.761855 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls podName:3c568fe3-f52b-46e4-97ed-d385287f6a7c nodeName:}" failed. No retries permitted until 2026-07-02 17:33:01.261833929 +0000 UTC m=+2168.451150942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls") pod "maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" (UID: "3c568fe3-f52b-46e4-97ed-d385287f6a7c") : secret "maas-api-e2e-shared-a-73dc61-serving-cert" not found Jul 02 17:33:00.762355 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.762336 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c568fe3-f52b-46e4-97ed-d385287f6a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:00.774034 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:00.774006 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgpq\" (UniqueName: \"kubernetes.io/projected/3c568fe3-f52b-46e4-97ed-d385287f6a7c-kube-api-access-5lgpq\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:01.267584 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:01.267554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:01.267769 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:33:01.267691 2582 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-73dc61-serving-cert: secret "maas-api-e2e-shared-a-73dc61-serving-cert" not found Jul 02 17:33:01.267769 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:33:01.267766 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls podName:3c568fe3-f52b-46e4-97ed-d385287f6a7c nodeName:}" failed. No retries permitted until 2026-07-02 17:33:02.267748139 +0000 UTC m=+2169.457065151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls") pod "maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" (UID: "3c568fe3-f52b-46e4-97ed-d385287f6a7c") : secret "maas-api-e2e-shared-a-73dc61-serving-cert" not found Jul 02 17:33:02.278641 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:02.278605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:02.281148 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:02.281124 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:02.426974 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:02.426933 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:02.576787 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:02.576761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2"] Jul 02 17:33:02.579048 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:33:02.579016 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c568fe3_f52b_46e4_97ed_d385287f6a7c.slice/crio-37c6f790c1606967038003eeb795d1f3c5485baffc075810eb865f11a7035d4e WatchSource:0}: Error finding container 37c6f790c1606967038003eeb795d1f3c5485baffc075810eb865f11a7035d4e: Status 404 returned error can't find the container with id 37c6f790c1606967038003eeb795d1f3c5485baffc075810eb865f11a7035d4e Jul 02 17:33:02.947904 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:02.947866 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" event={"ID":"3c568fe3-f52b-46e4-97ed-d385287f6a7c","Type":"ContainerStarted","Data":"37c6f790c1606967038003eeb795d1f3c5485baffc075810eb865f11a7035d4e"} Jul 02 17:33:03.953654 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:03.953620 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" event={"ID":"3c568fe3-f52b-46e4-97ed-d385287f6a7c","Type":"ContainerStarted","Data":"c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98"} Jul 02 17:33:03.954115 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:03.953766 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:03.970722 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:03.970675 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" podStartSLOduration=3.5266698229999998 podStartE2EDuration="3.970659162s" totalImagesPullingTime="443.989339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:33:03.969219307 +0000 UTC m=+2171.158536339" watchObservedRunningTime="2026-07-02 17:33:03.970659162 +0000 UTC m=+2171.159976193" Jul 02 17:33:08.778894 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:08.778858 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl"] Jul 02 17:33:08.783052 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:08.783028 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:08.785941 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:08.785916 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-b85f5a-serving-cert\"" Jul 02 17:33:08.798204 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:08.795655 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl"] Jul 02 17:33:08.941473 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:08.941417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqrr\" (UniqueName: \"kubernetes.io/projected/5eb5696a-8c27-4d68-9f93-912aecad0d43-kube-api-access-zlqrr\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:08.941473 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:08.941482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb5696a-8c27-4d68-9f93-912aecad0d43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:08.941723 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:08.941585 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.042866 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.042767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqrr\" (UniqueName: \"kubernetes.io/projected/5eb5696a-8c27-4d68-9f93-912aecad0d43-kube-api-access-zlqrr\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.042866 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.042826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb5696a-8c27-4d68-9f93-912aecad0d43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.042866 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.042862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.043146 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:33:09.042979 2582 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-b85f5a-serving-cert: secret "maas-api-e2e-shared-b-b85f5a-serving-cert" not found Jul 02 17:33:09.043146 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:33:09.043048 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls podName:5eb5696a-8c27-4d68-9f93-912aecad0d43 nodeName:}" failed. No retries permitted until 2026-07-02 17:33:09.543027853 +0000 UTC m=+2176.732344867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls") pod "maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" (UID: "5eb5696a-8c27-4d68-9f93-912aecad0d43") : secret "maas-api-e2e-shared-b-b85f5a-serving-cert" not found Jul 02 17:33:09.043589 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.043569 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb5696a-8c27-4d68-9f93-912aecad0d43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.051187 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.051152 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqrr\" (UniqueName: \"kubernetes.io/projected/5eb5696a-8c27-4d68-9f93-912aecad0d43-kube-api-access-zlqrr\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.548048 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.548015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.550449 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.550415 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls\") pod \"maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.703934 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.703894 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:09.840385 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.840355 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl"] Jul 02 17:33:09.843234 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:33:09.843206 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb5696a_8c27_4d68_9f93_912aecad0d43.slice/crio-c3b4f5a0c10de3dca079127f43ea5707e4f61c19fc06ebe9b1fcc5a067f4c5b1 WatchSource:0}: Error finding container c3b4f5a0c10de3dca079127f43ea5707e4f61c19fc06ebe9b1fcc5a067f4c5b1: Status 404 returned error can't find the container with id c3b4f5a0c10de3dca079127f43ea5707e4f61c19fc06ebe9b1fcc5a067f4c5b1 Jul 02 17:33:09.964209 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.964180 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:33:09.984422 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:09.984390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" event={"ID":"5eb5696a-8c27-4d68-9f93-912aecad0d43","Type":"ContainerStarted","Data":"c3b4f5a0c10de3dca079127f43ea5707e4f61c19fc06ebe9b1fcc5a067f4c5b1"} Jul 02 17:33:10.990125 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:10.990085 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" event={"ID":"5eb5696a-8c27-4d68-9f93-912aecad0d43","Type":"ContainerStarted","Data":"d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c"} Jul 02 17:33:10.990615 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:10.990182 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:11.006863 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:11.006805 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" podStartSLOduration=2.670884045 podStartE2EDuration="3.006785397s" totalImagesPullingTime="335.901352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:33:11.006061732 +0000 UTC m=+2178.195378778" watchObservedRunningTime="2026-07-02 17:33:11.006785397 +0000 UTC m=+2178.196102418" Jul 02 17:33:16.999848 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:16.999816 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:33:21.282526 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.282494 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw"] Jul 02 17:33:21.285849 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.285826 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.289221 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.289197 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-73dc61\"/\"auth-test-model-73dc61-kserve-self-signed-certs\"" Jul 02 17:33:21.289347 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.289225 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-73dc61\"/\"kube-root-ca.crt\"" Jul 02 17:33:21.289347 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.289314 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-73dc61\"/\"openshift-service-ca.crt\"" Jul 02 17:33:21.289347 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.289323 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-73dc61\"/\"default-dockercfg-psnz4\"" Jul 02 17:33:21.297117 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.297093 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw"] Jul 02 17:33:21.451636 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.451598 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2977058-8dd0-4dd9-9b42-e32670d28664-tls-certs\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.451841 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.451648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-model-cache\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.451841 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.451716 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjpm\" (UniqueName: \"kubernetes.io/projected/d2977058-8dd0-4dd9-9b42-e32670d28664-kube-api-access-tqjpm\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.451841 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.451790 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-dshm\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.451841 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.451825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-tmp-dir\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.451986 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.451854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-home\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.552778 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.552674 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-dshm\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.552778 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.552720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-tmp-dir\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.552778 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.552774 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-home\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.553059 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.552803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2977058-8dd0-4dd9-9b42-e32670d28664-tls-certs\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.553059 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.552841 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-model-cache\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.553059 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.552885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjpm\" (UniqueName: \"kubernetes.io/projected/d2977058-8dd0-4dd9-9b42-e32670d28664-kube-api-access-tqjpm\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.553222 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.553173 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-tmp-dir\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.553283 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.553233 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-home\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.553283 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.553252 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-model-cache\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.555071 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.555051 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-dshm\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.555202 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.555184 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2977058-8dd0-4dd9-9b42-e32670d28664-tls-certs\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.560888 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.560866 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjpm\" (UniqueName: \"kubernetes.io/projected/d2977058-8dd0-4dd9-9b42-e32670d28664-kube-api-access-tqjpm\") pod \"auth-test-model-73dc61-kserve-f9986c47-9fjfw\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.594810 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.594779 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:21.725225 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:21.725194 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw"] Jul 02 17:33:21.727568 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:33:21.727542 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2977058_8dd0_4dd9_9b42_e32670d28664.slice/crio-f77c6e932df4a953fce88258fd50218e75b5358a53e92c8453082df5d771552c WatchSource:0}: Error finding container f77c6e932df4a953fce88258fd50218e75b5358a53e92c8453082df5d771552c: Status 404 returned error can't find the container with id f77c6e932df4a953fce88258fd50218e75b5358a53e92c8453082df5d771552c Jul 02 17:33:22.042468 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:22.042426 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" event={"ID":"d2977058-8dd0-4dd9-9b42-e32670d28664","Type":"ContainerStarted","Data":"96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd"} Jul 02 17:33:22.042656 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:22.042476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" event={"ID":"d2977058-8dd0-4dd9-9b42-e32670d28664","Type":"ContainerStarted","Data":"f77c6e932df4a953fce88258fd50218e75b5358a53e92c8453082df5d771552c"} Jul 02 17:33:22.059033 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:22.058982 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" podStartSLOduration=1.058965406 podStartE2EDuration="1.058965406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:33:22.057534464 +0000 UTC m=+2189.246851496" watchObservedRunningTime="2026-07-02 17:33:22.058965406 +0000 UTC m=+2189.248282556" Jul 02 17:33:31.595863 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:31.595827 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:31.596360 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:31.596105 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:31.608485 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:31.608459 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:32.099574 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:32.099548 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:33:43.183092 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.183059 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r"] Jul 02 17:33:43.186672 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.186651 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.189342 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.189322 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b85f5a\"/\"openshift-service-ca.crt\"" Jul 02 17:33:43.190146 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.190127 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b85f5a\"/\"kube-root-ca.crt\"" Jul 02 17:33:43.190221 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.190150 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b85f5a\"/\"default-dockercfg-cv65r\"" Jul 02 17:33:43.190221 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.190207 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b85f5a\"/\"auth-test-model-b85f5a-kserve-self-signed-certs\"" Jul 02 17:33:43.197307 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.197283 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r"] Jul 02 17:33:43.235095 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.235066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-dshm\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.235095 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.235096 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwlv\" (UniqueName: \"kubernetes.io/projected/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-kube-api-access-hzwlv\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.235249 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.235125 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tmp-dir\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.235249 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.235163 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-model-cache\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.235249 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.235206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-home\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.235352 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.235283 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tls-certs\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.336489 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.336456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tls-certs\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.336489 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.336491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-dshm\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.336695 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.336509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwlv\" (UniqueName: \"kubernetes.io/projected/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-kube-api-access-hzwlv\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.336695 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.336533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tmp-dir\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.336695 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.336548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-model-cache\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.336695 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.336595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-home\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.337046 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.337017 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tmp-dir\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.337046 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.337045 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-home\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.337231 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.337068 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-model-cache\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.338822 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.338805 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-dshm\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.339112 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.339092 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tls-certs\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.344387 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.344360 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwlv\" (UniqueName: \"kubernetes.io/projected/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-kube-api-access-hzwlv\") pod \"auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.497122 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.497026 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:43.625825 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:43.625793 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r"] Jul 02 17:33:43.627911 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:33:43.627883 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22ed8d4f_cfbb_4d36_bb25_b89ebb2ec057.slice/crio-249d13d28feae6b5c49dc0fbef1aa1c78d99c2b86de826e49ca9c15f993c166d WatchSource:0}: Error finding container 249d13d28feae6b5c49dc0fbef1aa1c78d99c2b86de826e49ca9c15f993c166d: Status 404 returned error can't find the container with id 249d13d28feae6b5c49dc0fbef1aa1c78d99c2b86de826e49ca9c15f993c166d Jul 02 17:33:44.144921 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:44.144880 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" event={"ID":"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057","Type":"ContainerStarted","Data":"c8ded9f421007a2e78eab61155b54e112f0116641f82a7bda2f766b2ba135aff"} Jul 02 17:33:44.145119 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:44.144933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" event={"ID":"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057","Type":"ContainerStarted","Data":"249d13d28feae6b5c49dc0fbef1aa1c78d99c2b86de826e49ca9c15f993c166d"} Jul 02 17:33:44.160904 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:44.160854 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" podStartSLOduration=1.160839818 podStartE2EDuration="1.160839818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:33:44.159841127 +0000 UTC m=+2211.349158184" watchObservedRunningTime="2026-07-02 17:33:44.160839818 +0000 UTC m=+2211.350156849" Jul 02 17:33:53.497846 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:53.497808 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:53.498252 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:53.497869 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:53.510589 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:53.510563 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:33:54.201048 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:33:54.201021 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:35:33.425981 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:33.425950 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw"] Jul 02 17:35:34.489251 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:34.489215 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r"] Jul 02 17:35:48.445682 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.445561 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" podUID="d2977058-8dd0-4dd9-9b42-e32670d28664" containerName="main" containerID="cri-o://96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd" gracePeriod=45 Jul 02 17:35:48.698332 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.698265 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:35:48.728023 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.727987 2582 generic.go:356] "Generic (PLEG): container finished" podID="d2977058-8dd0-4dd9-9b42-e32670d28664" containerID="96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd" exitCode=0 Jul 02 17:35:48.728180 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.728081 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" Jul 02 17:35:48.728180 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.728112 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" event={"ID":"d2977058-8dd0-4dd9-9b42-e32670d28664","Type":"ContainerDied","Data":"96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd"} Jul 02 17:35:48.728180 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.728151 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw" event={"ID":"d2977058-8dd0-4dd9-9b42-e32670d28664","Type":"ContainerDied","Data":"f77c6e932df4a953fce88258fd50218e75b5358a53e92c8453082df5d771552c"} Jul 02 17:35:48.728180 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.728167 2582 scope.go:117] "RemoveContainer" containerID="96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd" Jul 02 17:35:48.736495 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.736478 2582 scope.go:117] "RemoveContainer" containerID="96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd" Jul 02 17:35:48.736801 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:35:48.736783 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd\": container with ID starting with 96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd not found: ID does not exist" containerID="96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd" Jul 02 17:35:48.736861 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.736811 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd"} err="failed to get container status \"96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd\": rpc error: code = NotFound desc = could not find container \"96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd\": container with ID starting with 96efd1dbd052ba8de7d152a94abdd7ece579c6ddfb5d559d159e107105338bfd not found: ID does not exist" Jul 02 17:35:48.819950 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.819914 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-home\") pod \"d2977058-8dd0-4dd9-9b42-e32670d28664\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " Jul 02 17:35:48.820138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.819956 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-model-cache\") pod \"d2977058-8dd0-4dd9-9b42-e32670d28664\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " Jul 02 17:35:48.820138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.819987 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-dshm\") pod \"d2977058-8dd0-4dd9-9b42-e32670d28664\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " Jul 02 17:35:48.820138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820046 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2977058-8dd0-4dd9-9b42-e32670d28664-tls-certs\") pod \"d2977058-8dd0-4dd9-9b42-e32670d28664\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " Jul 02 17:35:48.820138 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820067 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-tmp-dir\") pod \"d2977058-8dd0-4dd9-9b42-e32670d28664\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " Jul 02 17:35:48.820367 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820218 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-home" (OuterVolumeSpecName: "home") pod "d2977058-8dd0-4dd9-9b42-e32670d28664" (UID: "d2977058-8dd0-4dd9-9b42-e32670d28664"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:35:48.820367 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820229 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqjpm\" (UniqueName: \"kubernetes.io/projected/d2977058-8dd0-4dd9-9b42-e32670d28664-kube-api-access-tqjpm\") pod \"d2977058-8dd0-4dd9-9b42-e32670d28664\" (UID: \"d2977058-8dd0-4dd9-9b42-e32670d28664\") " Jul 02 17:35:48.820367 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820279 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2977058-8dd0-4dd9-9b42-e32670d28664" (UID: "d2977058-8dd0-4dd9-9b42-e32670d28664"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:35:48.820367 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820352 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2977058-8dd0-4dd9-9b42-e32670d28664" (UID: "d2977058-8dd0-4dd9-9b42-e32670d28664"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:35:48.820585 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820566 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-home\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:48.820626 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820587 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-model-cache\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:48.820626 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.820597 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-tmp-dir\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:48.822332 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.822315 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2977058-8dd0-4dd9-9b42-e32670d28664-kube-api-access-tqjpm" (OuterVolumeSpecName: "kube-api-access-tqjpm") pod "d2977058-8dd0-4dd9-9b42-e32670d28664" (UID: "d2977058-8dd0-4dd9-9b42-e32670d28664"). InnerVolumeSpecName "kube-api-access-tqjpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:35:48.822566 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.822551 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-dshm" (OuterVolumeSpecName: "dshm") pod "d2977058-8dd0-4dd9-9b42-e32670d28664" (UID: "d2977058-8dd0-4dd9-9b42-e32670d28664"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:35:48.822613 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.822580 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2977058-8dd0-4dd9-9b42-e32670d28664-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2977058-8dd0-4dd9-9b42-e32670d28664" (UID: "d2977058-8dd0-4dd9-9b42-e32670d28664"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:35:48.921517 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.921479 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2977058-8dd0-4dd9-9b42-e32670d28664-tls-certs\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:48.921517 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.921515 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjpm\" (UniqueName: \"kubernetes.io/projected/d2977058-8dd0-4dd9-9b42-e32670d28664-kube-api-access-tqjpm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:48.921758 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:48.921529 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2977058-8dd0-4dd9-9b42-e32670d28664-dshm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:49.049260 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.049223 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw"] Jul 02 17:35:49.052467 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.052442 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/auth-test-model-73dc61-kserve-f9986c47-9fjfw"] Jul 02 17:35:49.265566 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.265524 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2977058-8dd0-4dd9-9b42-e32670d28664" path="/var/lib/kubelet/pods/d2977058-8dd0-4dd9-9b42-e32670d28664/volumes" Jul 02 17:35:49.512323 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.512267 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" podUID="22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" containerName="main" containerID="cri-o://c8ded9f421007a2e78eab61155b54e112f0116641f82a7bda2f766b2ba135aff" gracePeriod=45 Jul 02 17:35:49.734546 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.734518 2582 generic.go:356] "Generic (PLEG): container finished" podID="22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" containerID="c8ded9f421007a2e78eab61155b54e112f0116641f82a7bda2f766b2ba135aff" exitCode=0 Jul 02 17:35:49.734712 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.734599 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" event={"ID":"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057","Type":"ContainerDied","Data":"c8ded9f421007a2e78eab61155b54e112f0116641f82a7bda2f766b2ba135aff"} Jul 02 17:35:49.765977 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.765917 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:35:49.930501 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930463 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tmp-dir\") pod \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " Jul 02 17:35:49.930501 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930503 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-home\") pod \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " Jul 02 17:35:49.930869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930521 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-dshm\") pod \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " Jul 02 17:35:49.930869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930540 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tls-certs\") pod \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " Jul 02 17:35:49.930869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930596 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzwlv\" (UniqueName: \"kubernetes.io/projected/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-kube-api-access-hzwlv\") pod \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " Jul 02 17:35:49.930869 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930614 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-model-cache\") pod \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\" (UID: \"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057\") " Jul 02 17:35:49.931080 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930875 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-home" (OuterVolumeSpecName: "home") pod "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" (UID: "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:35:49.931080 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930888 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" (UID: "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:35:49.931080 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.930974 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-model-cache" (OuterVolumeSpecName: "model-cache") pod "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" (UID: "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:35:49.932852 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.932823 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-dshm" (OuterVolumeSpecName: "dshm") pod "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" (UID: "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:35:49.932975 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.932894 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-kube-api-access-hzwlv" (OuterVolumeSpecName: "kube-api-access-hzwlv") pod "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" (UID: "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057"). InnerVolumeSpecName "kube-api-access-hzwlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:35:49.932975 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:49.932927 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" (UID: "22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:35:50.032006 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.031917 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tmp-dir\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:50.032006 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.031946 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-home\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:50.032006 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.031954 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-dshm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:50.032006 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.031962 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-tls-certs\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:50.032006 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.031970 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwlv\" (UniqueName: \"kubernetes.io/projected/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-kube-api-access-hzwlv\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:50.032006 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.031978 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057-model-cache\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:35:50.742166 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.742132 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" event={"ID":"22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057","Type":"ContainerDied","Data":"249d13d28feae6b5c49dc0fbef1aa1c78d99c2b86de826e49ca9c15f993c166d"} Jul 02 17:35:50.742663 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.742177 2582 scope.go:117] "RemoveContainer" containerID="c8ded9f421007a2e78eab61155b54e112f0116641f82a7bda2f766b2ba135aff" Jul 02 17:35:50.742663 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.742142 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r" Jul 02 17:35:50.762905 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.762880 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r"] Jul 02 17:35:50.764420 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:50.764400 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/auth-test-model-b85f5a-kserve-7c79f86d87-vzk9r"] Jul 02 17:35:51.265832 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:51.265791 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" path="/var/lib/kubelet/pods/22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057/volumes" Jul 02 17:35:54.158552 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.158510 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r"] Jul 02 17:35:54.159151 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.159125 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2977058-8dd0-4dd9-9b42-e32670d28664" containerName="main" Jul 02 17:35:54.159151 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.159151 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2977058-8dd0-4dd9-9b42-e32670d28664" containerName="main" Jul 02 17:35:54.159324 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.159192 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" containerName="main" Jul 02 17:35:54.159324 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.159200 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" containerName="main" Jul 02 17:35:54.159324 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.159295 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ed8d4f-cfbb-4d36-bb25-b89ebb2ec057" containerName="main" Jul 02 17:35:54.159324 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.159313 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2977058-8dd0-4dd9-9b42-e32670d28664" containerName="main" Jul 02 17:35:54.163946 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.163929 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.166285 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.166264 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b85f5a\"/\"kube-root-ca.crt\"" Jul 02 17:35:54.167142 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.167125 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b85f5a\"/\"openshift-service-ca.crt\"" Jul 02 17:35:54.167245 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.167126 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b85f5a\"/\"default-dockercfg-cv65r\"" Jul 02 17:35:54.167245 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.167165 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b85f5a\"/\"sub-test-model-b85f5a-kserve-self-signed-certs\"" Jul 02 17:35:54.171133 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.171113 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r"] Jul 02 17:35:54.269145 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.269112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-dshm\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.269145 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.269144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-home\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.269350 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.269202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-model-cache\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.269350 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.269299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e359575c-ee96-4051-8621-38d03e2ce1bb-tls-certs\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.269350 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.269332 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb58n\" (UniqueName: \"kubernetes.io/projected/e359575c-ee96-4051-8621-38d03e2ce1bb-kube-api-access-wb58n\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.269449 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.269362 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-tmp-dir\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370333 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb58n\" (UniqueName: \"kubernetes.io/projected/e359575c-ee96-4051-8621-38d03e2ce1bb-kube-api-access-wb58n\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370502 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-tmp-dir\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370502 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-dshm\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370502 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-home\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370502 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370471 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-model-cache\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370684 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e359575c-ee96-4051-8621-38d03e2ce1bb-tls-certs\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370793 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370773 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-tmp-dir\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370858 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370827 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-home\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.370955 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.370937 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-model-cache\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.372761 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.372740 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-dshm\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.372930 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.372914 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e359575c-ee96-4051-8621-38d03e2ce1bb-tls-certs\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.377932 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.377913 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb58n\" (UniqueName: \"kubernetes.io/projected/e359575c-ee96-4051-8621-38d03e2ce1bb-kube-api-access-wb58n\") pod \"sub-test-model-b85f5a-kserve-75c48b9977-wl84r\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.474207 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.474126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:35:54.604116 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.604087 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r"] Jul 02 17:35:54.605592 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:35:54.605563 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode359575c_ee96_4051_8621_38d03e2ce1bb.slice/crio-baecd7db61eef2c6595d7c4b5f8f75e3b1f90a025ed6902d64a4cb8ae6d8aa7f WatchSource:0}: Error finding container baecd7db61eef2c6595d7c4b5f8f75e3b1f90a025ed6902d64a4cb8ae6d8aa7f: Status 404 returned error can't find the container with id baecd7db61eef2c6595d7c4b5f8f75e3b1f90a025ed6902d64a4cb8ae6d8aa7f Jul 02 17:35:54.763194 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.763110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" event={"ID":"e359575c-ee96-4051-8621-38d03e2ce1bb","Type":"ContainerStarted","Data":"8b1900f266d95a501b8aef811557487652d2d65a7a05e60d76be55ec1b0168b0"} Jul 02 17:35:54.763194 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:35:54.763150 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" event={"ID":"e359575c-ee96-4051-8621-38d03e2ce1bb","Type":"ContainerStarted","Data":"baecd7db61eef2c6595d7c4b5f8f75e3b1f90a025ed6902d64a4cb8ae6d8aa7f"} Jul 02 17:36:04.475342 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:04.475297 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:36:04.475860 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:04.475358 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:36:04.487509 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:04.487479 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:36:04.502836 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:04.502792 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" podStartSLOduration=10.502778329 podStartE2EDuration="10.502778329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:35:54.77832469 +0000 UTC m=+2341.967641721" watchObservedRunningTime="2026-07-02 17:36:04.502778329 +0000 UTC m=+2351.692095360" Jul 02 17:36:04.822293 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:04.822211 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:36:35.883692 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:35.883660 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r"] Jul 02 17:36:47.196007 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.195977 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2"] Jul 02 17:36:47.199622 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.199602 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.202153 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.202130 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-73dc61\"/\"openshift-service-ca.crt\"" Jul 02 17:36:47.202262 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.202213 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-73dc61\"/\"default-dockercfg-psnz4\"" Jul 02 17:36:47.203339 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.203323 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-73dc61\"/\"kube-root-ca.crt\"" Jul 02 17:36:47.203429 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.203412 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-73dc61\"/\"rate-test-model-73dc61-kserve-self-signed-certs\"" Jul 02 17:36:47.214742 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.214709 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2"] Jul 02 17:36:47.339975 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.339931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13332b3e-fb58-4b64-9e84-39df2ab59bff-tls-certs\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.340158 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.339985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-home\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.340158 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.340031 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvrc\" (UniqueName: \"kubernetes.io/projected/13332b3e-fb58-4b64-9e84-39df2ab59bff-kube-api-access-nvvrc\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.340158 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.340105 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-tmp-dir\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.340276 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.340193 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-dshm\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.340276 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.340231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-model-cache\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.441419 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441382 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13332b3e-fb58-4b64-9e84-39df2ab59bff-tls-certs\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.441419 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-home\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.441652 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvrc\" (UniqueName: \"kubernetes.io/projected/13332b3e-fb58-4b64-9e84-39df2ab59bff-kube-api-access-nvvrc\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.441652 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-tmp-dir\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.441652 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-dshm\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.441652 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-model-cache\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.441916 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441891 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-home\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.442007 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441954 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-tmp-dir\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.442007 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.441972 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-model-cache\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.443818 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.443787 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-dshm\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.444002 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.443982 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13332b3e-fb58-4b64-9e84-39df2ab59bff-tls-certs\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.449715 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.449656 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvrc\" (UniqueName: \"kubernetes.io/projected/13332b3e-fb58-4b64-9e84-39df2ab59bff-kube-api-access-nvvrc\") pod \"rate-test-model-73dc61-kserve-684975dc47-jsfq2\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.509674 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.509641 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:47.640346 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:47.640321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2"] Jul 02 17:36:47.642757 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:36:47.642703 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13332b3e_fb58_4b64_9e84_39df2ab59bff.slice/crio-caa59789bccbe136718a5feb389e2367f40694ba534e9a39d321aa2b5c25e6e4 WatchSource:0}: Error finding container caa59789bccbe136718a5feb389e2367f40694ba534e9a39d321aa2b5c25e6e4: Status 404 returned error can't find the container with id caa59789bccbe136718a5feb389e2367f40694ba534e9a39d321aa2b5c25e6e4 Jul 02 17:36:48.015675 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:48.015585 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" event={"ID":"13332b3e-fb58-4b64-9e84-39df2ab59bff","Type":"ContainerStarted","Data":"7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9"} Jul 02 17:36:48.015675 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:48.015634 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" event={"ID":"13332b3e-fb58-4b64-9e84-39df2ab59bff","Type":"ContainerStarted","Data":"caa59789bccbe136718a5feb389e2367f40694ba534e9a39d321aa2b5c25e6e4"} Jul 02 17:36:48.032298 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:48.032246 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" podStartSLOduration=1.032232246 podStartE2EDuration="1.032232246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:36:48.031286236 +0000 UTC m=+2395.220603268" watchObservedRunningTime="2026-07-02 17:36:48.032232246 +0000 UTC m=+2395.221549281" Jul 02 17:36:50.903757 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:50.903703 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" podUID="e359575c-ee96-4051-8621-38d03e2ce1bb" containerName="main" containerID="cri-o://8b1900f266d95a501b8aef811557487652d2d65a7a05e60d76be55ec1b0168b0" gracePeriod=45 Jul 02 17:36:51.032283 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.032254 2582 generic.go:356] "Generic (PLEG): container finished" podID="e359575c-ee96-4051-8621-38d03e2ce1bb" containerID="8b1900f266d95a501b8aef811557487652d2d65a7a05e60d76be55ec1b0168b0" exitCode=0 Jul 02 17:36:51.032414 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.032326 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" event={"ID":"e359575c-ee96-4051-8621-38d03e2ce1bb","Type":"ContainerDied","Data":"8b1900f266d95a501b8aef811557487652d2d65a7a05e60d76be55ec1b0168b0"} Jul 02 17:36:51.160978 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.160923 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:36:51.278586 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.278562 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-model-cache\") pod \"e359575c-ee96-4051-8621-38d03e2ce1bb\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " Jul 02 17:36:51.278782 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.278593 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-tmp-dir\") pod \"e359575c-ee96-4051-8621-38d03e2ce1bb\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " Jul 02 17:36:51.278782 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.278615 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-home\") pod \"e359575c-ee96-4051-8621-38d03e2ce1bb\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " Jul 02 17:36:51.278782 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.278652 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb58n\" (UniqueName: \"kubernetes.io/projected/e359575c-ee96-4051-8621-38d03e2ce1bb-kube-api-access-wb58n\") pod \"e359575c-ee96-4051-8621-38d03e2ce1bb\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " Jul 02 17:36:51.278782 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.278707 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e359575c-ee96-4051-8621-38d03e2ce1bb-tls-certs\") pod \"e359575c-ee96-4051-8621-38d03e2ce1bb\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " Jul 02 17:36:51.278782 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.278763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-dshm\") pod \"e359575c-ee96-4051-8621-38d03e2ce1bb\" (UID: \"e359575c-ee96-4051-8621-38d03e2ce1bb\") " Jul 02 17:36:51.279073 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.279030 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e359575c-ee96-4051-8621-38d03e2ce1bb" (UID: "e359575c-ee96-4051-8621-38d03e2ce1bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:36:51.279073 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.279054 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e359575c-ee96-4051-8621-38d03e2ce1bb" (UID: "e359575c-ee96-4051-8621-38d03e2ce1bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:36:51.279073 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.279067 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-home" (OuterVolumeSpecName: "home") pod "e359575c-ee96-4051-8621-38d03e2ce1bb" (UID: "e359575c-ee96-4051-8621-38d03e2ce1bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:36:51.280955 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.280926 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e359575c-ee96-4051-8621-38d03e2ce1bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e359575c-ee96-4051-8621-38d03e2ce1bb" (UID: "e359575c-ee96-4051-8621-38d03e2ce1bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:36:51.280955 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.280932 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-dshm" (OuterVolumeSpecName: "dshm") pod "e359575c-ee96-4051-8621-38d03e2ce1bb" (UID: "e359575c-ee96-4051-8621-38d03e2ce1bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:36:51.281094 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.281023 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e359575c-ee96-4051-8621-38d03e2ce1bb-kube-api-access-wb58n" (OuterVolumeSpecName: "kube-api-access-wb58n") pod "e359575c-ee96-4051-8621-38d03e2ce1bb" (UID: "e359575c-ee96-4051-8621-38d03e2ce1bb"). InnerVolumeSpecName "kube-api-access-wb58n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:36:51.379774 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.379719 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e359575c-ee96-4051-8621-38d03e2ce1bb-tls-certs\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:36:51.379774 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.379769 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-dshm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:36:51.379774 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.379777 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-model-cache\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:36:51.380029 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.379802 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-tmp-dir\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:36:51.380029 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.379810 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e359575c-ee96-4051-8621-38d03e2ce1bb-home\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:36:51.380029 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:51.379819 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb58n\" (UniqueName: \"kubernetes.io/projected/e359575c-ee96-4051-8621-38d03e2ce1bb-kube-api-access-wb58n\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:36:52.038377 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:52.038344 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" event={"ID":"e359575c-ee96-4051-8621-38d03e2ce1bb","Type":"ContainerDied","Data":"baecd7db61eef2c6595d7c4b5f8f75e3b1f90a025ed6902d64a4cb8ae6d8aa7f"} Jul 02 17:36:52.038377 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:52.038375 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r" Jul 02 17:36:52.038915 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:52.038393 2582 scope.go:117] "RemoveContainer" containerID="8b1900f266d95a501b8aef811557487652d2d65a7a05e60d76be55ec1b0168b0" Jul 02 17:36:52.059565 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:52.059536 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r"] Jul 02 17:36:52.063000 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:52.062981 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b85f5a/sub-test-model-b85f5a-kserve-75c48b9977-wl84r"] Jul 02 17:36:53.267803 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:53.267771 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e359575c-ee96-4051-8621-38d03e2ce1bb" path="/var/lib/kubelet/pods/e359575c-ee96-4051-8621-38d03e2ce1bb/volumes" Jul 02 17:36:53.386035 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:53.386011 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:36:53.400240 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:53.400214 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2d9jh_51136f16-4d0a-41fa-b9d6-3cb9f16fd3bd/ovn-acl-logging/0.log" Jul 02 17:36:57.510637 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:57.510596 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:57.511166 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:57.510664 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:57.523501 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:57.523473 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:36:58.079698 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:36:58.079670 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:37:55.074268 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:37:55.074226 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2"] Jul 02 17:38:10.094247 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.094184 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" podUID="13332b3e-fb58-4b64-9e84-39df2ab59bff" containerName="main" containerID="cri-o://7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9" gracePeriod=45 Jul 02 17:38:10.347833 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.347102 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:38:10.398797 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.398765 2582 generic.go:356] "Generic (PLEG): container finished" podID="13332b3e-fb58-4b64-9e84-39df2ab59bff" containerID="7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9" exitCode=0 Jul 02 17:38:10.398996 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.398827 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" Jul 02 17:38:10.398996 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.398854 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" event={"ID":"13332b3e-fb58-4b64-9e84-39df2ab59bff","Type":"ContainerDied","Data":"7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9"} Jul 02 17:38:10.398996 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.398911 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2" event={"ID":"13332b3e-fb58-4b64-9e84-39df2ab59bff","Type":"ContainerDied","Data":"caa59789bccbe136718a5feb389e2367f40694ba534e9a39d321aa2b5c25e6e4"} Jul 02 17:38:10.398996 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.398935 2582 scope.go:117] "RemoveContainer" containerID="7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9" Jul 02 17:38:10.407139 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.407125 2582 scope.go:117] "RemoveContainer" containerID="7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9" Jul 02 17:38:10.407378 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:38:10.407362 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9\": container with ID starting with 7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9 not found: ID does not exist" containerID="7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9" Jul 02 17:38:10.407427 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.407387 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9"} err="failed to get container status \"7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9\": rpc error: code = NotFound desc = could not find container \"7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9\": container with ID starting with 7da26d60c65ceaba1e0984922188f7c2e7087aaee79cca3903af3690e19cf7c9 not found: ID does not exist" Jul 02 17:38:10.471996 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.471971 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-dshm\") pod \"13332b3e-fb58-4b64-9e84-39df2ab59bff\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " Jul 02 17:38:10.472140 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472018 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-home\") pod \"13332b3e-fb58-4b64-9e84-39df2ab59bff\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " Jul 02 17:38:10.472140 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472074 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-tmp-dir\") pod \"13332b3e-fb58-4b64-9e84-39df2ab59bff\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " Jul 02 17:38:10.472140 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472109 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-model-cache\") pod \"13332b3e-fb58-4b64-9e84-39df2ab59bff\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " Jul 02 17:38:10.472140 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472131 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvvrc\" (UniqueName: \"kubernetes.io/projected/13332b3e-fb58-4b64-9e84-39df2ab59bff-kube-api-access-nvvrc\") pod \"13332b3e-fb58-4b64-9e84-39df2ab59bff\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " Jul 02 17:38:10.472324 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472197 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13332b3e-fb58-4b64-9e84-39df2ab59bff-tls-certs\") pod \"13332b3e-fb58-4b64-9e84-39df2ab59bff\" (UID: \"13332b3e-fb58-4b64-9e84-39df2ab59bff\") " Jul 02 17:38:10.472366 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472316 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-home" (OuterVolumeSpecName: "home") pod "13332b3e-fb58-4b64-9e84-39df2ab59bff" (UID: "13332b3e-fb58-4b64-9e84-39df2ab59bff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:38:10.472366 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472331 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13332b3e-fb58-4b64-9e84-39df2ab59bff" (UID: "13332b3e-fb58-4b64-9e84-39df2ab59bff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:38:10.472466 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472403 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-model-cache" (OuterVolumeSpecName: "model-cache") pod "13332b3e-fb58-4b64-9e84-39df2ab59bff" (UID: "13332b3e-fb58-4b64-9e84-39df2ab59bff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:38:10.472549 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472528 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-home\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:38:10.472611 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472552 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-tmp-dir\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:38:10.472611 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.472567 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-model-cache\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:38:10.474230 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.474204 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-dshm" (OuterVolumeSpecName: "dshm") pod "13332b3e-fb58-4b64-9e84-39df2ab59bff" (UID: "13332b3e-fb58-4b64-9e84-39df2ab59bff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 17:38:10.474338 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.474257 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13332b3e-fb58-4b64-9e84-39df2ab59bff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13332b3e-fb58-4b64-9e84-39df2ab59bff" (UID: "13332b3e-fb58-4b64-9e84-39df2ab59bff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:38:10.474382 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.474330 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13332b3e-fb58-4b64-9e84-39df2ab59bff-kube-api-access-nvvrc" (OuterVolumeSpecName: "kube-api-access-nvvrc") pod "13332b3e-fb58-4b64-9e84-39df2ab59bff" (UID: "13332b3e-fb58-4b64-9e84-39df2ab59bff"). InnerVolumeSpecName "kube-api-access-nvvrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:38:10.573620 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.573583 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13332b3e-fb58-4b64-9e84-39df2ab59bff-dshm\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:38:10.573620 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.573613 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvrc\" (UniqueName: \"kubernetes.io/projected/13332b3e-fb58-4b64-9e84-39df2ab59bff-kube-api-access-nvvrc\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:38:10.573856 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.573628 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13332b3e-fb58-4b64-9e84-39df2ab59bff-tls-certs\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:38:10.723338 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.723302 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2"] Jul 02 17:38:10.728039 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:10.728013 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-73dc61/rate-test-model-73dc61-kserve-684975dc47-jsfq2"] Jul 02 17:38:11.266114 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:38:11.266082 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13332b3e-fb58-4b64-9e84-39df2ab59bff" path="/var/lib/kubelet/pods/13332b3e-fb58-4b64-9e84-39df2ab59bff/volumes" Jul 02 17:39:06.822500 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:06.822466 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2"] Jul 02 17:39:06.823016 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:06.822663 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" podUID="3c568fe3-f52b-46e4-97ed-d385287f6a7c" containerName="maas-api" containerID="cri-o://c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98" gracePeriod=30 Jul 02 17:39:07.061915 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.061885 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:39:07.145459 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.145434 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c568fe3-f52b-46e4-97ed-d385287f6a7c-openshift-service-ca\") pod \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " Jul 02 17:39:07.145626 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.145484 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls\") pod \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " Jul 02 17:39:07.145626 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.145519 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lgpq\" (UniqueName: \"kubernetes.io/projected/3c568fe3-f52b-46e4-97ed-d385287f6a7c-kube-api-access-5lgpq\") pod \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\" (UID: \"3c568fe3-f52b-46e4-97ed-d385287f6a7c\") " Jul 02 17:39:07.145918 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.145889 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c568fe3-f52b-46e4-97ed-d385287f6a7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c568fe3-f52b-46e4-97ed-d385287f6a7c" (UID: "3c568fe3-f52b-46e4-97ed-d385287f6a7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:39:07.147721 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.147697 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c568fe3-f52b-46e4-97ed-d385287f6a7c" (UID: "3c568fe3-f52b-46e4-97ed-d385287f6a7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:39:07.147858 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.147753 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c568fe3-f52b-46e4-97ed-d385287f6a7c-kube-api-access-5lgpq" (OuterVolumeSpecName: "kube-api-access-5lgpq") pod "3c568fe3-f52b-46e4-97ed-d385287f6a7c" (UID: "3c568fe3-f52b-46e4-97ed-d385287f6a7c"). InnerVolumeSpecName "kube-api-access-5lgpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:39:07.246487 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.246461 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c568fe3-f52b-46e4-97ed-d385287f6a7c-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:39:07.246487 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.246486 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lgpq\" (UniqueName: \"kubernetes.io/projected/3c568fe3-f52b-46e4-97ed-d385287f6a7c-kube-api-access-5lgpq\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:39:07.246487 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.246496 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c568fe3-f52b-46e4-97ed-d385287f6a7c-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:39:07.661146 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.661105 2582 generic.go:356] "Generic (PLEG): container finished" podID="3c568fe3-f52b-46e4-97ed-d385287f6a7c" containerID="c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98" exitCode=0 Jul 02 17:39:07.661349 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.661153 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" Jul 02 17:39:07.661349 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.661194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" event={"ID":"3c568fe3-f52b-46e4-97ed-d385287f6a7c","Type":"ContainerDied","Data":"c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98"} Jul 02 17:39:07.661349 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.661242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2" event={"ID":"3c568fe3-f52b-46e4-97ed-d385287f6a7c","Type":"ContainerDied","Data":"37c6f790c1606967038003eeb795d1f3c5485baffc075810eb865f11a7035d4e"} Jul 02 17:39:07.661349 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.661266 2582 scope.go:117] "RemoveContainer" containerID="c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98" Jul 02 17:39:07.669692 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.669675 2582 scope.go:117] "RemoveContainer" containerID="c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98" Jul 02 17:39:07.669929 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:39:07.669910 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98\": container with ID starting with c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98 not found: ID does not exist" containerID="c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98" Jul 02 17:39:07.669999 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.669942 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98"} err="failed to get container status \"c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98\": rpc error: code = NotFound desc = could not find container \"c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98\": container with ID starting with c1defba362894f963663abae6d01dacab1ad5de4e3c249a15316bc0bfafbbf98 not found: ID does not exist" Jul 02 17:39:07.679451 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.679424 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2"] Jul 02 17:39:07.683811 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:07.683784 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-73dc61-f87c9b4d7-5n9s2"] Jul 02 17:39:09.266206 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:09.266166 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c568fe3-f52b-46e4-97ed-d385287f6a7c" path="/var/lib/kubelet/pods/3c568fe3-f52b-46e4-97ed-d385287f6a7c/volumes" Jul 02 17:39:14.000948 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.000918 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl"] Jul 02 17:39:14.001377 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.001140 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" podUID="5eb5696a-8c27-4d68-9f93-912aecad0d43" containerName="maas-api" containerID="cri-o://d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c" gracePeriod=30 Jul 02 17:39:14.262090 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.262039 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:39:14.406993 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.406948 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb5696a-8c27-4d68-9f93-912aecad0d43-openshift-service-ca\") pod \"5eb5696a-8c27-4d68-9f93-912aecad0d43\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " Jul 02 17:39:14.412494 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.407265 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlqrr\" (UniqueName: \"kubernetes.io/projected/5eb5696a-8c27-4d68-9f93-912aecad0d43-kube-api-access-zlqrr\") pod \"5eb5696a-8c27-4d68-9f93-912aecad0d43\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " Jul 02 17:39:14.412494 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.407370 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls\") pod \"5eb5696a-8c27-4d68-9f93-912aecad0d43\" (UID: \"5eb5696a-8c27-4d68-9f93-912aecad0d43\") " Jul 02 17:39:14.412494 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.407602 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb5696a-8c27-4d68-9f93-912aecad0d43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5eb5696a-8c27-4d68-9f93-912aecad0d43" (UID: "5eb5696a-8c27-4d68-9f93-912aecad0d43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 17:39:14.412494 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.407775 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb5696a-8c27-4d68-9f93-912aecad0d43-openshift-service-ca\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:39:14.412494 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.409909 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb5696a-8c27-4d68-9f93-912aecad0d43-kube-api-access-zlqrr" (OuterVolumeSpecName: "kube-api-access-zlqrr") pod "5eb5696a-8c27-4d68-9f93-912aecad0d43" (UID: "5eb5696a-8c27-4d68-9f93-912aecad0d43"). InnerVolumeSpecName "kube-api-access-zlqrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 17:39:14.412841 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.412549 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5eb5696a-8c27-4d68-9f93-912aecad0d43" (UID: "5eb5696a-8c27-4d68-9f93-912aecad0d43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 17:39:14.508308 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.508279 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eb5696a-8c27-4d68-9f93-912aecad0d43-maas-api-tls\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:39:14.508308 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.508307 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqrr\" (UniqueName: \"kubernetes.io/projected/5eb5696a-8c27-4d68-9f93-912aecad0d43-kube-api-access-zlqrr\") on node \"ip-10-0-134-244.ec2.internal\" DevicePath \"\"" Jul 02 17:39:14.699646 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.699605 2582 generic.go:356] "Generic (PLEG): container finished" podID="5eb5696a-8c27-4d68-9f93-912aecad0d43" containerID="d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c" exitCode=0 Jul 02 17:39:14.699646 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.699637 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" event={"ID":"5eb5696a-8c27-4d68-9f93-912aecad0d43","Type":"ContainerDied","Data":"d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c"} Jul 02 17:39:14.699889 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.699683 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" Jul 02 17:39:14.699889 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.699713 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl" event={"ID":"5eb5696a-8c27-4d68-9f93-912aecad0d43","Type":"ContainerDied","Data":"c3b4f5a0c10de3dca079127f43ea5707e4f61c19fc06ebe9b1fcc5a067f4c5b1"} Jul 02 17:39:14.699889 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.699752 2582 scope.go:117] "RemoveContainer" containerID="d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c" Jul 02 17:39:14.708173 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.708157 2582 scope.go:117] "RemoveContainer" containerID="d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c" Jul 02 17:39:14.708441 ip-10-0-134-244 kubenswrapper[2582]: E0702 17:39:14.708422 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c\": container with ID starting with d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c not found: ID does not exist" containerID="d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c" Jul 02 17:39:14.708488 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.708450 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c"} err="failed to get container status \"d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c\": rpc error: code = NotFound desc = could not find container \"d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c\": container with ID starting with d53fb676fabfc09c9569ec4a56d4dd776a0c4dbd5c283e8d578947435a06c62c not found: ID does not exist" Jul 02 17:39:14.721189 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.721169 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl"] Jul 02 17:39:14.722785 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:14.722768 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b85f5a-fbc6dd677-pgdgl"] Jul 02 17:39:15.266104 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:15.266069 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eb5696a-8c27-4d68-9f93-912aecad0d43" path="/var/lib/kubelet/pods/5eb5696a-8c27-4d68-9f93-912aecad0d43/volumes" Jul 02 17:39:20.938430 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:20.938391 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bb8b8694c-2sqvd_3ab7866b-0b49-4646-9e07-041ea86b56d8/authorino/0.log" Jul 02 17:39:26.148942 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:26.148910 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29716875-hqmcf_7f94ad41-99b8-4f34-82f9-b884d0016656/cleanup/0.log" Jul 02 17:39:26.262784 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:26.262755 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29716890-rwj8r_2b9d30e1-64d5-4579-90a9-fd0a9c459b14/cleanup/0.log" Jul 02 17:39:26.389823 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:26.389775 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dfcd6c9c4-qwn6v_2d164e70-5e2d-413d-999a-771b4e7b7881/manager/0.log" Jul 02 17:39:26.514762 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:26.514670 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zd7v5_37f119d3-97a0-476f-bb44-31e80a93d991/server/0.log" Jul 02 17:39:26.637073 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:26.637038 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qngj2_869652f0-e49a-4195-8cbf-7367b9599b38/manager/0.log" Jul 02 17:39:26.756548 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:26.756521 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8b6c9f55f-82825_adb297bd-54ae-429d-97ed-061bad4f5635/copy-manifests/0.log" Jul 02 17:39:26.764062 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:26.764036 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8b6c9f55f-82825_adb297bd-54ae-429d-97ed-061bad4f5635/manager/0.log" Jul 02 17:39:28.480021 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:28.479987 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bb8b8694c-2sqvd_3ab7866b-0b49-4646-9e07-041ea86b56d8/authorino/0.log" Jul 02 17:39:29.928938 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:29.928910 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5785f89f-fqhjp_44b6c6d7-667b-4faa-b1e6-4a033e0a0032/kube-auth-proxy/0.log" Jul 02 17:39:30.340880 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:30.340797 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59f49fd4f6-25l5g_804c7c00-4f66-49a7-8219-bece72c63475/payload-pre-processing/0.log" Jul 02 17:39:30.470361 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:30.470329 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bbcb654f4-w468k_041b935c-fe2b-460e-a549-cc0189cf0963/payload-processing/0.log" Jul 02 17:39:31.117661 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:31.117634 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-nw6vj_14e8de8b-6ca2-4445-8977-cc4dd5cc7ba9/main/0.log" Jul 02 17:39:31.239568 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:31.239532 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vxktn_abdc36a7-6bf4-4486-83c3-328c070a9f08/main/0.log" Jul 02 17:39:39.207294 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:39.207262 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rn2wj_8ffdd0d1-7f10-427b-8ed5-5b877f73840e/global-pull-secret-syncer/0.log" Jul 02 17:39:39.375609 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:39.375576 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bx746_36ccebec-913b-4330-8a79-43988fae2af6/konnectivity-agent/0.log" Jul 02 17:39:39.483775 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:39.483671 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-244.ec2.internal_282f21947ce88ed32d4f969496593dd1/haproxy/0.log" Jul 02 17:39:44.748307 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:44.748272 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bb8b8694c-2sqvd_3ab7866b-0b49-4646-9e07-041ea86b56d8/authorino/0.log" Jul 02 17:39:46.905457 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:46.905416 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-n4jww_d30eed73-7ea8-4ead-9662-764a01687571/kube-state-metrics/0.log" Jul 02 17:39:46.921660 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:46.921637 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-n4jww_d30eed73-7ea8-4ead-9662-764a01687571/kube-rbac-proxy-main/0.log" Jul 02 17:39:46.942803 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:46.942783 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-n4jww_d30eed73-7ea8-4ead-9662-764a01687571/kube-rbac-proxy-self/0.log" Jul 02 17:39:47.107589 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:47.107559 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcpm4_facaaa7e-843e-427e-a094-fee3bfedd82f/node-exporter/0.log" Jul 02 17:39:47.140207 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:47.140186 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcpm4_facaaa7e-843e-427e-a094-fee3bfedd82f/kube-rbac-proxy/0.log" Jul 02 17:39:47.186581 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:47.186511 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcpm4_facaaa7e-843e-427e-a094-fee3bfedd82f/init-textfile/0.log" Jul 02 17:39:48.972124 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972086 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z"] Jul 02 17:39:48.972505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972404 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c568fe3-f52b-46e4-97ed-d385287f6a7c" containerName="maas-api" Jul 02 17:39:48.972505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972416 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c568fe3-f52b-46e4-97ed-d385287f6a7c" containerName="maas-api" Jul 02 17:39:48.972505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972422 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13332b3e-fb58-4b64-9e84-39df2ab59bff" containerName="main" Jul 02 17:39:48.972505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972429 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="13332b3e-fb58-4b64-9e84-39df2ab59bff" containerName="main" Jul 02 17:39:48.972505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972437 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb5696a-8c27-4d68-9f93-912aecad0d43" containerName="maas-api" Jul 02 17:39:48.972505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972443 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb5696a-8c27-4d68-9f93-912aecad0d43" containerName="maas-api" Jul 02 17:39:48.972505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972459 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e359575c-ee96-4051-8621-38d03e2ce1bb" containerName="main" Jul 02 17:39:48.972505 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972467 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e359575c-ee96-4051-8621-38d03e2ce1bb" containerName="main" Jul 02 17:39:48.972800 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972522 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb5696a-8c27-4d68-9f93-912aecad0d43" containerName="maas-api" Jul 02 17:39:48.972800 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972532 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e359575c-ee96-4051-8621-38d03e2ce1bb" containerName="main" Jul 02 17:39:48.972800 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972538 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="13332b3e-fb58-4b64-9e84-39df2ab59bff" containerName="main" Jul 02 17:39:48.972800 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.972544 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c568fe3-f52b-46e4-97ed-d385287f6a7c" containerName="maas-api" Jul 02 17:39:48.975494 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.975478 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:48.977906 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.977886 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-55pg4\"/\"kube-root-ca.crt\"" Jul 02 17:39:48.978704 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.978690 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-55pg4\"/\"openshift-service-ca.crt\"" Jul 02 17:39:48.978704 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.978697 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-55pg4\"/\"default-dockercfg-9kgtg\"" Jul 02 17:39:48.984908 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:48.984886 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z"] Jul 02 17:39:49.010598 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.010569 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-5w7nd_e405fd52-1c06-449f-a1ed-c8f0437ec471/networking-console-plugin/0.log" Jul 02 17:39:49.101792 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.101755 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-lib-modules\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.101970 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.101815 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-proc\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.101970 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.101835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hsqx\" (UniqueName: \"kubernetes.io/projected/63939b43-9800-44db-9638-82687260506f-kube-api-access-9hsqx\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.101970 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.101878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-podres\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.102079 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.101958 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-sys\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.202675 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-sys\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.202675 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202680 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-lib-modules\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.202903 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-proc\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.202903 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hsqx\" (UniqueName: \"kubernetes.io/projected/63939b43-9800-44db-9638-82687260506f-kube-api-access-9hsqx\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.202903 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202789 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-sys\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.202903 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202859 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-proc\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.202903 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-podres\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.202903 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202880 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-lib-modules\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.203108 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.202964 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63939b43-9800-44db-9638-82687260506f-podres\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.210657 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.210617 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hsqx\" (UniqueName: \"kubernetes.io/projected/63939b43-9800-44db-9638-82687260506f-kube-api-access-9hsqx\") pod \"perf-node-gather-daemonset-tbf9z\" (UID: \"63939b43-9800-44db-9638-82687260506f\") " pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.285648 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.285574 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.414058 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.414028 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z"] Jul 02 17:39:49.416618 ip-10-0-134-244 kubenswrapper[2582]: W0702 17:39:49.416591 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod63939b43_9800_44db_9638_82687260506f.slice/crio-823c49d716cac9d3935eff955701c9121449678cd018bc1350bb38d8a7c531cd WatchSource:0}: Error finding container 823c49d716cac9d3935eff955701c9121449678cd018bc1350bb38d8a7c531cd: Status 404 returned error can't find the container with id 823c49d716cac9d3935eff955701c9121449678cd018bc1350bb38d8a7c531cd Jul 02 17:39:49.863882 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.863848 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" event={"ID":"63939b43-9800-44db-9638-82687260506f","Type":"ContainerStarted","Data":"e50132cee9dc3e49996952c746a3c6a05a8abf8f9f76fa29f5e135d1548d28a3"} Jul 02 17:39:49.864100 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.863902 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" event={"ID":"63939b43-9800-44db-9638-82687260506f","Type":"ContainerStarted","Data":"823c49d716cac9d3935eff955701c9121449678cd018bc1350bb38d8a7c531cd"} Jul 02 17:39:49.864100 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.863988 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:49.879928 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:49.879873 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" podStartSLOduration=1.879855143 podStartE2EDuration="1.879855143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 17:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:39:49.878172566 +0000 UTC m=+2577.067489608" watchObservedRunningTime="2026-07-02 17:39:49.879855143 +0000 UTC m=+2577.069172173" Jul 02 17:39:51.851779 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:51.851747 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n99hl_0edd7266-95a1-45ad-bc52-caca04bd572f/dns/0.log" Jul 02 17:39:51.874878 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:51.874848 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n99hl_0edd7266-95a1-45ad-bc52-caca04bd572f/kube-rbac-proxy/0.log" Jul 02 17:39:51.962129 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:51.962093 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pfmsz_dc678e74-bdb7-47dd-8302-5cfc86a9864b/dns-node-resolver/0.log" Jul 02 17:39:52.534170 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:52.534135 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bwj5n_ab2a2297-66d2-4a77-8e4c-b07cbe46725c/node-ca/0.log" Jul 02 17:39:53.710002 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:53.709971 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5785f89f-fqhjp_44b6c6d7-667b-4faa-b1e6-4a033e0a0032/kube-auth-proxy/0.log" Jul 02 17:39:53.865601 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:53.865518 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59f49fd4f6-25l5g_804c7c00-4f66-49a7-8219-bece72c63475/payload-pre-processing/0.log" Jul 02 17:39:53.951019 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:53.950988 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bbcb654f4-w468k_041b935c-fe2b-460e-a549-cc0189cf0963/payload-processing/0.log" Jul 02 17:39:54.612542 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:54.612507 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6vwwl_ec2ef2cf-1d58-4d0f-83bb-3434f314b777/serve-healthcheck-canary/0.log" Jul 02 17:39:55.124071 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:55.124042 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-s2ss9_b402035e-12c0-41f9-94e8-79ae8d9e4d63/insights-operator/0.log" Jul 02 17:39:55.147289 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:55.147262 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wdnp_35992b57-3bd9-42c2-8287-c50dd2ddc6a9/kube-rbac-proxy/0.log" Jul 02 17:39:55.168568 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:55.168543 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wdnp_35992b57-3bd9-42c2-8287-c50dd2ddc6a9/exporter/0.log" Jul 02 17:39:55.188884 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:55.188857 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wdnp_35992b57-3bd9-42c2-8287-c50dd2ddc6a9/extractor/0.log" Jul 02 17:39:55.878188 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:55.878161 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-55pg4/perf-node-gather-daemonset-tbf9z" Jul 02 17:39:57.743197 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:57.743169 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29716875-hqmcf_7f94ad41-99b8-4f34-82f9-b884d0016656/cleanup/0.log" Jul 02 17:39:57.766417 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:57.766388 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29716890-rwj8r_2b9d30e1-64d5-4579-90a9-fd0a9c459b14/cleanup/0.log" Jul 02 17:39:57.933571 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:57.933533 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dfcd6c9c4-qwn6v_2d164e70-5e2d-413d-999a-771b4e7b7881/manager/0.log" Jul 02 17:39:57.952692 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:57.952663 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zd7v5_37f119d3-97a0-476f-bb44-31e80a93d991/server/0.log" Jul 02 17:39:58.056667 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:58.056590 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qngj2_869652f0-e49a-4195-8cbf-7367b9599b38/manager/0.log" Jul 02 17:39:58.078024 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:58.077996 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8b6c9f55f-82825_adb297bd-54ae-429d-97ed-061bad4f5635/manager/0.log" Jul 02 17:39:58.095472 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:58.095453 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8b6c9f55f-82825_adb297bd-54ae-429d-97ed-061bad4f5635/copy-manifests/0.log" Jul 02 17:39:59.978614 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:39:59.978530 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f677bc8d7-5cvff_7825283b-118e-4100-ba7e-07ea67e51d1f/manager/0.log" Jul 02 17:40:00.031053 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:00.031022 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-w4qdb_10f3e190-ba80-4420-97c2-0fe12039bf0b/openshift-lws-operator/0.log" Jul 02 17:40:00.105598 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:00.105573 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5785f89f-fqhjp_44b6c6d7-667b-4faa-b1e6-4a033e0a0032/kube-auth-proxy/0.log" Jul 02 17:40:00.239593 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:00.239507 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59f49fd4f6-25l5g_804c7c00-4f66-49a7-8219-bece72c63475/payload-pre-processing/0.log" Jul 02 17:40:00.307829 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:00.307802 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bbcb654f4-w468k_041b935c-fe2b-460e-a549-cc0189cf0963/payload-processing/0.log" Jul 02 17:40:01.780495 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:01.780469 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5785f89f-fqhjp_44b6c6d7-667b-4faa-b1e6-4a033e0a0032/kube-auth-proxy/0.log" Jul 02 17:40:01.920367 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:01.920336 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59f49fd4f6-25l5g_804c7c00-4f66-49a7-8219-bece72c63475/payload-pre-processing/0.log" Jul 02 17:40:01.992252 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:01.992211 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bbcb654f4-w468k_041b935c-fe2b-460e-a549-cc0189cf0963/payload-processing/0.log" Jul 02 17:40:06.450655 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:06.450576 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zrq4_e1e076a9-2f26-4006-ad54-cd2d108d34de/kube-multus-additional-cni-plugins/0.log" Jul 02 17:40:06.470271 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:06.470245 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zrq4_e1e076a9-2f26-4006-ad54-cd2d108d34de/egress-router-binary-copy/0.log" Jul 02 17:40:06.488620 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:06.488597 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zrq4_e1e076a9-2f26-4006-ad54-cd2d108d34de/cni-plugins/0.log" Jul 02 17:40:06.507316 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:06.507280 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zrq4_e1e076a9-2f26-4006-ad54-cd2d108d34de/bond-cni-plugin/0.log" Jul 02 17:40:06.529493 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:06.529458 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zrq4_e1e076a9-2f26-4006-ad54-cd2d108d34de/routeoverride-cni/0.log" Jul 02 17:40:06.547878 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:06.547851 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zrq4_e1e076a9-2f26-4006-ad54-cd2d108d34de/whereabouts-cni-bincopy/0.log" Jul 02 17:40:06.567004 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:06.566975 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zrq4_e1e076a9-2f26-4006-ad54-cd2d108d34de/whereabouts-cni/0.log" Jul 02 17:40:06.937872 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:06.937838 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cnkgs_ed13545a-affc-489d-a843-24e0e2344de6/kube-multus/0.log" Jul 02 17:40:07.071570 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:07.071538 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tr57l_0535e4ec-90d9-47eb-9cd6-9f1e9225a103/network-metrics-daemon/0.log" Jul 02 17:40:07.094753 ip-10-0-134-244 kubenswrapper[2582]: I0702 17:40:07.094702 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tr57l_0535e4ec-90d9-47eb-9cd6-9f1e9225a103/kube-rbac-proxy/0.log"