Jul 06 16:26:27.871004 ip-10-0-142-118 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 16:26:27.871024 ip-10-0-142-118 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 16:26:27.871035 ip-10-0-142-118 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 16:26:27.871369 ip-10-0-142-118 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 16:26:38.043446 ip-10-0-142-118 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 16:26:38.043478 ip-10-0-142-118 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 24d00a9b28294eb2b2db98d7e1eec056 --
Jul 06 16:28:41.237925 ip-10-0-142-118 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 16:28:41.618491 ip-10-0-142-118 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 16:28:41.618491 ip-10-0-142-118 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 16:28:41.618491 ip-10-0-142-118 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 16:28:41.618491 ip-10-0-142-118 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 16:28:41.618491 ip-10-0-142-118 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 16:28:41.620324 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.620228    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 16:28:41.627806 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627782    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 16:28:41.627806 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627805    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 16:28:41.627806 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627808    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 16:28:41.627806 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627811    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627815    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627819    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627822    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627826    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627828    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627831    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627837    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627840    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627843    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627846    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627849    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627852    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627854    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627857    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627860    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627862    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627865    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627868    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627870    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 16:28:41.627948 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627873    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627876    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627879    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627882    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627884    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627886    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627889    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627891    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627894    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627896    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627898    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627901    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627904    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627907    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627910    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627912    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627916    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627920    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627924    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 16:28:41.628424 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627927    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627932    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627935    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627938    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627941    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627943    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627946    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627948    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627951    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627954    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627956    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627959    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627962    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627964    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627967    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627969    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627972    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627974    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627977    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 16:28:41.628909 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627980    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627982    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627984    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627987    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627989    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627991    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627994    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.627997    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628000    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628002    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628004    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628007    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628010    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628013    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628015    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628018    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628020    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628022    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628025    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 16:28:41.629372 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628028    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628030    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628032    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628035    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628037    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628040    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628042    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628044    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628047    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628049    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628052    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628055    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628058    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628060    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628063    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628065    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628067    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628070    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628073    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628820    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 16:28:41.629845 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628828    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628831    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628834    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628836    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628839    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628842    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628845    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628848    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628850    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628853    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628855    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628857    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628860    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628862    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628865    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628867    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628870    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628872    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628875    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628877    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 16:28:41.630316 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628880    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628882    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628884    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628887    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628890    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628893    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628895    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628900    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628904    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628906    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628909    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628912    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628917    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628920    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628923    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628925    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628928    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628930    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628933    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 16:28:41.630866 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628936    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628939    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628942    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628944    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628947    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628950    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628953    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628956    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628958    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628961    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628963    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628965    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628968    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628970    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628973    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628977    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628980    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628983    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628985    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 16:28:41.631340 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628988    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628990    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628993    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628995    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.628997    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629000    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629003    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629005    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629008    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629010    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629012    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629015    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629017    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629021    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629025    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629028    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629031    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629034    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629036    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629040    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 16:28:41.631842 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629042    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629045    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629047    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629050    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629053    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629055    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629058    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629060    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629063    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629065    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629068    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629070    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629072    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629075    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629077    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629079    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629081    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629084    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629087    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629089    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 16:28:41.632321 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629492    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629497    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629500    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629502    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629505    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629508    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629512    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629514    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629517    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629520    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629522    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629525    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629527    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629530    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629533    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629535    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629537    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629540    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629542    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629545    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 16:28:41.632826 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629547    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629550    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629552    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629555    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629559    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629562    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629565    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629568    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629571    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629576    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629579    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629582    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629584    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629587    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629589    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629592    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629594    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629597    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629600    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629618    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 16:28:41.633312 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629621    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629624    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629626    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629629    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629631    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629634    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629636    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629638    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629641    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629644    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629646    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629649    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629651    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629655    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629658    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629661    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629663    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629667    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629669    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 16:28:41.633830 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629672    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629675    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629679    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629681    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629684    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629686    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629690    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629693    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629696    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629699    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629703    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629706    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629708    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629711    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629714    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629716    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629719    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629721    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629724    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 16:28:41.634284 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629726    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629729    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629731    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629734    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629736    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629738    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629741    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629744    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629746    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629748    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629751    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629753    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629756    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629758    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629761    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629765    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629767    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629769    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629772    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629774    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 16:28:41.634769 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.629777    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629869    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629889    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629900    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629905    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629909    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629912    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629925    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629933    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629937    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629940    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629943    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629947    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629950    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629953    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629956    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629959    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629962    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629965    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629970    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629972    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629975    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629978    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629981    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 16:28:41.635261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629985    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629988    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629991    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629995    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.629998    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630001    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630004    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630007    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630010    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630014    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630017    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630021    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630024    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630027    2578 flags.go:64] FLAG: --enable-server="true"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630030    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630034    2578 flags.go:64] FLAG: --event-burst="100"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630038    2578 flags.go:64] FLAG: --event-qps="50"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630041    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630044    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630046    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630050    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630053    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630056    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630059    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630062    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 16:28:41.635888 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630064    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630067    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630070    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630073    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630075    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630078    2578 flags.go:64] FLAG: --feature-gates=""
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630081    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630084    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630087    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630090    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630094    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630097    2578 flags.go:64] FLAG: --help="false"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630100    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-118.ec2.internal"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630104    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630107    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630109    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630113    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630116    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630120    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630123    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630126    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630128    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630131    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630134    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 16:28:41.636495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630138    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630141    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630144    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630147    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630149    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630152    2578 flags.go:64] FLAG: --lock-file=""
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630155    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630158    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630160    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630166    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630169    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630172    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630174    2578 flags.go:64] FLAG: --logging-format="text"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630177    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630180    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630183    2578 flags.go:64] FLAG: --manifest-url=""
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630186    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630190    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630193    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630198    2578 flags.go:64] FLAG: --max-pods="110"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630200    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630203    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630206    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630209    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630212    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 16:28:41.637092 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630215    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630219    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630229    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630232    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630235    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630238    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630240    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630246    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630250    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630253    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630256    2578 flags.go:64] FLAG: --port="10250"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630259    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630262    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0432b81d3b302c24d"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630265    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630268    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630271    2578 flags.go:64] FLAG: --register-node="true"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630273    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630277    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630279    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630282    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630285    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630289    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630292    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630295    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630298    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 16:28:41.637744 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630300    2578 flags.go:64] FLAG: --runonce="false"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630303    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630306    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630309    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630312    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630314    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630318    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630321    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630325    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630329    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630331    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630334    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630337    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630340    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630343    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630345    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630351    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630354    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630356    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630361    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630364    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630366    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630369    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630371    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630374    2578 flags.go:64] FLAG: --v="2"
Jul 06 16:28:41.638350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630378    2578 flags.go:64] FLAG: --version="false"
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630382    2578 flags.go:64] FLAG: --vmodule=""
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630385    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.630388    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630530    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630534    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630538    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630540    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630543    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630546    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630549    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630552    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630554    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630559    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630562    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630567    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630570    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630573    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630576    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630578    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 16:28:41.639150 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630581    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630583    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630586    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630588    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630592    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630594    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630597    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630599    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630602    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630619    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630621    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630624    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630626    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630629    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630632    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630634    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630636    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630639    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630643    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630645    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 16:28:41.639974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630648    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630651    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630654    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630656    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630659    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630661    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630663    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630668    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630670    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630672    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630675    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630678    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630680    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630683    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630686    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630689    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630693    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630696    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630699    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630702    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 16:28:41.640475 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630704    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630706    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630709    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630711    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630714    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630716    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630719    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630723    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630726    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630729    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630735    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630737    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630740    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630743    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630745    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630747    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630750    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630752    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630755    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 16:28:41.640986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630759    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630761    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630764    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630766    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630769    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630772    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630774    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630777    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630779    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630782    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630784    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630787    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630790    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630792    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630795    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630797    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630800    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630802    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630804    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630807    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 16:28:41.641452 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630810    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630812    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630815    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.630818    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.631778    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.640139    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.640159    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.640177    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.640185    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640282    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640288    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640291    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640294    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640297    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 16:28:41.642057 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640301    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640304    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640306    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640310    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640312    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640314    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640317    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640320    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640322    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640325    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640329    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640333    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640336    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640339    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640342    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640344    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640347    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640349    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640352    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640354    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 16:28:41.642413 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640357    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640360    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640363    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640368    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640371    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640374    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640377    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640380    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640382    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640385    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640387    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640390    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640393    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640396    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640398    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640400    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640403    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640405    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640408    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 16:28:41.642977 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640411    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640415    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640419    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640422    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640424    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640427    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640429    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640432    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640434    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640437    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640439    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640442    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640444    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640448    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640450    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640453    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640456    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640458    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640461    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640464    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 16:28:41.643455 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640466    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640469    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640471    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640474    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640476    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640479    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640482    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640484    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640487    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640490    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640493    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640495    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640497    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640501    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640503    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640506    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640508    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640511    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640513    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640516    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 16:28:41.643966 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640518    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640521    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640523    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640526    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640528    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640531    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640534    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640536    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640539    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640541    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640544    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640546    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640548    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640551    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 16:28:41.644447 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640553    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.640559    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640708    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640713    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640716    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640719    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640722    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640725    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640728    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640730    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640733    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640735    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640738    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640741    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640744    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 16:28:41.644811 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640747    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640749    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640752    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640754    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640757    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640759    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640761    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640764    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640766    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640769    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640772    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640774    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640777    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640779    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640782    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640784    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640787    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640789    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640792    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640794    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 16:28:41.645175 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640796    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640799    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640801    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640803    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640806    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640808    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640811    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640813    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640816    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640818    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640820    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640823    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640827    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640841    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640845    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640848    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640851    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640853    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640856    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 16:28:41.645662 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640859    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640861    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640864    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640866    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640869    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640872    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640875    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640877    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640880    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640883    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640886    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640888    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640891    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640893    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640896    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640898    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640900    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640903    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640905    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640908    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 16:28:41.646127 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640911    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640914    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640916    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640918    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640921    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640923    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640926    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640931    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640934    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640937    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640939    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640942    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640944    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640947    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640950    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640952    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640955    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640957    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640960    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640962    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 16:28:41.646623 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640965    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640967    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640970    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640972    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640974    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640977    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:41.640980    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.640985    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.641110    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.645115    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.646756    2578 server.go:1019] "Starting client certificate rotation"
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.646875    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 16:28:41.647124 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.646917    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 16:28:41.649959 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.649942    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 16:28:41.653158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.653134    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 16:28:41.668923 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.668901    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 06 16:28:41.671368 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.671350    2578 log.go:25] "Validated CRI v1 image API"
Jul 06 16:28:41.672224 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.672207    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 16:28:41.675453 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.675427    2578 fs.go:135] Filesystem UUIDs: map[6555b2b1-3bd5-4a39-845c-b70221b7c765:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cf6df7b4-2688-4c6e-8d92-d4d601059214:/dev/nvme0n1p3]
Jul 06 16:28:41.675533 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.675453    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 16:28:41.678568 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.678550    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 16:28:41.680256 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.680148    2578 manager.go:216] Machine: {Timestamp:2026-07-06 16:28:41.679620525 +0000 UTC m=+0.325024970 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086789 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24e97d77bec823c0b353bb7cccc5bc SystemUUID:ec24e97d-77be-c823-c0b3-53bb7cccc5bc BootID:24d00a9b-2829-4eb2-b2db-98d7e1eec056 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:72:fe:55:09:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:72:fe:55:09:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:fb:66:54:c7:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 16:28:41.680256 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.680255    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 16:28:41.680350 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.680333    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 16:28:41.683857 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.683829    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 16:28:41.684007 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.683859    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 16:28:41.684780 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.684769    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 16:28:41.684814 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.684783    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 16:28:41.684814 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.684791    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 16:28:41.684814 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.684806    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 16:28:41.686710 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.686696    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 16:28:41.690486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.690469    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 16:28:41.690663 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.690653    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 16:28:41.690703 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.690688    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 16:28:41.690731 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.690707    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 16:28:41.690731 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.690719    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 16:28:41.690789 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.690751    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 06 16:28:41.690789 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.690764    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 16:28:41.691323 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.691305    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 16:28:41.691323 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.691325    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 16:28:41.695286 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.695200    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 16:28:41.697969 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.697952    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 16:28:41.698017 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.698007    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 16:28:41.699580 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699568    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699585    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699591    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699596    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699622    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699629    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699637    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699642    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699649    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 16:28:41.699655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699654    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 16:28:41.699887 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699663    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 16:28:41.699887 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.699668    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 16:28:41.700375 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.700344    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 16:28:41.700425 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.700344    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 16:28:41.701293 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.701281    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 16:28:41.701293 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.701294    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 16:28:41.703277 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.703260    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 16:28:41.704135 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.704117    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f78sp"
Jul 06 16:28:41.704974 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.704961    2578 server.go:1268] "Started kubelet"
Jul 06 16:28:41.705081 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.705026    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 16:28:41.705182 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.705138    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 16:28:41.705245 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.705207    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 16:28:41.705969 ip-10-0-142-118 systemd[1]: Started Kubernetes Kubelet.
Jul 06 16:28:41.708394 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.707041    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 16:28:41.708394 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.707720    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 16:28:41.711284 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.711266    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f78sp"
Jul 06 16:28:41.712593 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.712561    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 16:28:41.716422 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.716398    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 16:28:41.716422 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.716405    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 16:28:41.716575 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.716522    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:41.717315 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.717298    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 16:28:41.717614 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.717341    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 16:28:41.717722 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.717630    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 16:28:41.717775 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.717770    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 16:28:41.717823 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.717778    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 16:28:41.718187 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.718084    2578 factory.go:55] Registering systemd factory
Jul 06 16:28:41.718187 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.718100    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 06 16:28:41.718396 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.717403    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-118.ec2.internal.18bfc0376a8662cd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-118.ec2.internal,UID:ip-10-0-142-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-118.ec2.internal,},FirstTimestamp:2026-07-06 16:28:41.704932045 +0000 UTC m=+0.350336475,LastTimestamp:2026-07-06 16:28:41.704932045 +0000 UTC m=+0.350336475,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-118.ec2.internal,}"
Jul 06 16:28:41.718515 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.718487    2578 factory.go:153] Registering CRI-O factory
Jul 06 16:28:41.718515 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.718508    2578 factory.go:223] Registration of the crio container factory successfully
Jul 06 16:28:41.718593 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.718572    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 16:28:41.718655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.718599    2578 factory.go:103] Registering Raw factory
Jul 06 16:28:41.718655 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.718635    2578 manager.go:1196] Started watching for new ooms in manager
Jul 06 16:28:41.719091 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.719079    2578 manager.go:349] Starting recovery of all containers
Jul 06 16:28:41.726080 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.726053    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 16:28:41.728775 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.728747    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-118.ec2.internal\" not found" node="ip-10-0-142-118.ec2.internal"
Jul 06 16:28:41.730443 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.730419    2578 manager.go:354] Recovery completed
Jul 06 16:28:41.735231 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.735216    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 16:28:41.737345 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.737328    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 16:28:41.737431 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.737363    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 16:28:41.737431 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.737377    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientPID"
Jul 06 16:28:41.737950 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.737932    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 16:28:41.737994 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.737953    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 16:28:41.737994 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.737968    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 16:28:41.737994 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.737975    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 16:28:41.738856 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.738837    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 16:28:41.738895 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.738877    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 16:28:41.738923 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.738893    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientPID"
Jul 06 16:28:41.739304 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.739292    2578 policy_none.go:49] "None policy: Start"
Jul 06 16:28:41.739336 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.739309    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 16:28:41.739336 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.739323    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 16:28:41.740620 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.740596    2578 policy_none.go:47] "Start"
Jul 06 16:28:41.781202 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.781176    2578 manager.go:337] "Starting Device Plugin manager"
Jul 06 16:28:41.781306 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.781223    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 16:28:41.781306 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.781236    2578 server.go:87] "Starting device plugin registration server"
Jul 06 16:28:41.781538 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.781525    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 16:28:41.781594 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.781539    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 16:28:41.781657 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.781648    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 16:28:41.781756 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.781739    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 16:28:41.781756 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.781753    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 16:28:41.782350 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.782328    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 16:28:41.782439 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.782385    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:41.882216 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.882130    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 16:28:41.883785 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.883126    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 16:28:41.883785 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.883159    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 16:28:41.883785 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.883173    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientPID"
Jul 06 16:28:41.883785 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.883201    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-118.ec2.internal"
Jul 06 16:28:41.891073 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.891054    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-118.ec2.internal"
Jul 06 16:28:41.891153 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.891081    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-118.ec2.internal\": node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:41.896893 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.896854    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 16:28:41.898241 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.898226    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 16:28:41.898341 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.898260    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 16:28:41.898341 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.898280    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 16:28:41.898341 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.898319    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 16:28:41.900105 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.900084    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 16:28:41.908053 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:41.908029    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:41.998449 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.998418    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal"]
Jul 06 16:28:41.998602 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:41.998512    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 16:28:42.000587 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.000571    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 16:28:42.000698 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.000601    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 16:28:42.000698 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.000636    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientPID"
Jul 06 16:28:42.002091 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.002077    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 16:28:42.002231 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.002204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.002334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.002263    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 16:28:42.003189 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.003169    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 16:28:42.003189 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.003183    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 16:28:42.003331 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.003203    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 16:28:42.003331 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.003218    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientPID"
Jul 06 16:28:42.003331 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.003205    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 16:28:42.003331 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.003254    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientPID"
Jul 06 16:28:42.004550 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.004537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.004598 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.004566    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 16:28:42.005325 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.005309    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 16:28:42.005406 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.005334    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 16:28:42.005406 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.005347    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeHasSufficientPID"
Jul 06 16:28:42.008208 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:42.008192    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:42.031705 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:42.031675    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-118.ec2.internal\" not found" node="ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.036195 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:42.036175    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-118.ec2.internal\" not found" node="ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.108493 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:42.108460    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:42.118792 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.118767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971ca3d09392685f8bb4e145bd0a3e41-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal\" (UID: \"971ca3d09392685f8bb4e145bd0a3e41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.118886 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.118794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971ca3d09392685f8bb4e145bd0a3e41-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal\" (UID: \"971ca3d09392685f8bb4e145bd0a3e41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.118886 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.118809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a9545f43e4880450f1a48b17d8b749c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-118.ec2.internal\" (UID: \"9a9545f43e4880450f1a48b17d8b749c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.209277 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:42.209203    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:42.219588 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.219560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971ca3d09392685f8bb4e145bd0a3e41-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal\" (UID: \"971ca3d09392685f8bb4e145bd0a3e41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.219680 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.219592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971ca3d09392685f8bb4e145bd0a3e41-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal\" (UID: \"971ca3d09392685f8bb4e145bd0a3e41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.219680 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.219623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a9545f43e4880450f1a48b17d8b749c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-118.ec2.internal\" (UID: \"9a9545f43e4880450f1a48b17d8b749c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.219680 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.219647    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971ca3d09392685f8bb4e145bd0a3e41-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal\" (UID: \"971ca3d09392685f8bb4e145bd0a3e41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.219799 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.219672    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971ca3d09392685f8bb4e145bd0a3e41-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal\" (UID: \"971ca3d09392685f8bb4e145bd0a3e41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.219799 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.219649    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a9545f43e4880450f1a48b17d8b749c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-118.ec2.internal\" (UID: \"9a9545f43e4880450f1a48b17d8b749c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.309784 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:42.309742    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:42.336301 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.336270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:42.341770 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:42.341748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:43.618703 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:43.618660    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:43.719277 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:43.719221    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:43.819869 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:43.819832    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:43.853331 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:43.853307    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 16:28:43.853975 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:43.853448    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 16:28:43.853975 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:43.853494    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 16:28:43.920965 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:43.920934    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:43.928477 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:43.928428    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 16:23:42 +0000 UTC" deadline="2027-12-18 22:34:23.965390461 +0000 UTC"
Jul 06 16:28:43.928477 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:43.928468    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12726h5m40.036924944s"
Jul 06 16:28:44.009191 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.009116    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 16:28:44.021283 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:44.021256    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:44.122118 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:44.122088    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:44.188230 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:44.188194    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9545f43e4880450f1a48b17d8b749c.slice/crio-92e0043c94b66fc7f592b5d0d49729ad782b79e69d74f44bad8e1e423d264c3a WatchSource:0}: Error finding container 92e0043c94b66fc7f592b5d0d49729ad782b79e69d74f44bad8e1e423d264c3a: Status 404 returned error can't find the container with id 92e0043c94b66fc7f592b5d0d49729ad782b79e69d74f44bad8e1e423d264c3a
Jul 06 16:28:44.188431 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:44.188415    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971ca3d09392685f8bb4e145bd0a3e41.slice/crio-a0fa50acd020e5f836924176f6f310f1e93e67e90833d0c8c6c746f7e8745aa0 WatchSource:0}: Error finding container a0fa50acd020e5f836924176f6f310f1e93e67e90833d0c8c6c746f7e8745aa0: Status 404 returned error can't find the container with id a0fa50acd020e5f836924176f6f310f1e93e67e90833d0c8c6c746f7e8745aa0
Jul 06 16:28:44.192134 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.192119    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 16:28:44.222347 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:44.222310    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:44.322880 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:44.322858    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-118.ec2.internal\" not found"
Jul 06 16:28:44.347154 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.347128    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 16:28:44.424514 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.424465    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:44.437576 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.437551    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 16:28:44.438629 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.438615    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal"
Jul 06 16:28:44.447219 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.447197    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 16:28:44.900029 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.899999    2578 apiserver.go:52] "Watching apiserver"
Jul 06 16:28:44.914779 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.914547    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 16:28:44.915201 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.915176    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-phxxj","openshift-dns/node-resolver-8k9x7","openshift-image-registry/node-ca-smb7q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal","openshift-multus/multus-49bs7","openshift-network-diagnostics/network-check-target-skj5h","openshift-network-operator/iptables-alerter-czt6b","kube-system/global-pull-secret-syncer-slpn6","kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt","openshift-multus/multus-additional-cni-plugins-7p6kj","openshift-multus/network-metrics-daemon-692bf","openshift-ovn-kubernetes/ovnkube-node-j748n","kube-system/konnectivity-agent-ktlhf"]
Jul 06 16:28:44.917657 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.917629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:44.919328 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.919309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.920318 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.920297    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 16:28:44.920425 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.920297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9997n\""
Jul 06 16:28:44.920713 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.920687    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 16:28:44.920713 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.920702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 16:28:44.920941 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.920925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:44.921866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.921840    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 16:28:44.922171 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.922155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 16:28:44.922309 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.922296    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xsz8p\""
Jul 06 16:28:44.922476 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.922462    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 16:28:44.922623 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.922611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 16:28:44.922662 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.922619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:28:44.922736 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.922723    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 16:28:44.923265 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.923247    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 16:28:44.923474 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.923443    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 16:28:44.923573 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.923492    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 16:28:44.923696 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.923655    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 16:28:44.923772 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.923741    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l5vc4\""
Jul 06 16:28:44.923875 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.923857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.924991 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.924974    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 16:28:44.926532 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.925232    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tgwjf\""
Jul 06 16:28:44.926532 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.925343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 16:28:44.926532 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.925656    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 16:28:44.926532 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.926260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 16:28:44.926532 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.926464    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 16:28:44.926532 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.926464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h6q85\""
Jul 06 16:28:44.926532 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.926527    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 16:28:44.927083 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.927066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:44.927161 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:44.927128    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:28:44.927739 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.927725    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 16:28:44.928700 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.928670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:44.928776 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:44.928756    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:28:44.930567 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.930550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:44.931602 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.931582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:44.933321 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.933086    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 16:28:44.933321 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.933147    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 16:28:44.933321 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.933306    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 16:28:44.933518 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.933374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:44.933576 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.933534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vpqtm\""
Jul 06 16:28:44.933787 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.933770    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6n8bf\""
Jul 06 16:28:44.933862 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.933810    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 16:28:44.934096 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934078    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 16:28:44.934576 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:44.934896 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-systemd\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.934896 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovnkube-script-lib\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.934896 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-serviceca\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:44.934896 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ea25d04-2500-4c7b-b77d-b84c3ce92344-konnectivity-ca\") pod \"konnectivity-agent-ktlhf\" (UID: \"7ea25d04-2500-4c7b-b77d-b84c3ce92344\") " pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:28:44.934896 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-cnibin\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-hostroot\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-node-log\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovnkube-config\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.934984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-env-overrides\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-socket-dir-parent\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-multus-certs\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-registration-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-etc-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-cni-netd\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-netns\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-daemon-config\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.935198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2w2r\" (UniqueName: \"kubernetes.io/projected/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-kube-api-access-d2w2r\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovn-node-metrics-cert\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r5nm\" (UniqueName: \"kubernetes.io/projected/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-kube-api-access-2r5nm\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-system-cni-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-conf-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-socket-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-kubelet\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-systemd-units\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-log-socket\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxpvg\" (UniqueName: \"kubernetes.io/projected/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-kube-api-access-kxpvg\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ea25d04-2500-4c7b-b77d-b84c3ce92344-agent-certs\") pod \"konnectivity-agent-ktlhf\" (UID: \"7ea25d04-2500-4c7b-b77d-b84c3ce92344\") " pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-sys-fs\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-slash\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.935676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-var-lib-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-ovn\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-cni-bin\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-k8s-cni-cncf-io\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-cni-bin\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-kubelet\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-device-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-host\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-cni-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-os-release\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7lb\" (UniqueName: \"kubernetes.io/projected/39cb27f9-a0fb-49bd-9a81-0a48771a9770-kube-api-access-db7lb\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.935892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-run-netns\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:44.935994    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.936009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.936023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39cb27f9-a0fb-49bd-9a81-0a48771a9770-cni-binary-copy\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.936037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-cni-multus\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936790 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.936142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-etc-kubernetes\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:44.936790 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.936175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:44.936790 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.936494    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9k8c8\""
Jul 06 16:28:44.936790 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.936541    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 16:28:44.936906 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.936854    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 16:28:44.938136 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.938099    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 16:28:44.938252 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.938146    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 16:28:44.939043 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.939024    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p8nsr\""
Jul 06 16:28:44.940356 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.940333    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 16:28:44.966075 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.966049    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fdrfh"
Jul 06 16:28:44.974319 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:44.974294    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fdrfh"
Jul 06 16:28:45.027080 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.027050    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 16:28:45.037321 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e56b49-6747-48b3-891d-09f7547abfe4-tmp-dir\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysconfig\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-device-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-host\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-cni-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6007e8ce-157f-403c-9c0d-651547e29295-iptables-alerter-script\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037442    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-device-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-sys\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037466    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-host\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:45.037486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-lib-modules\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037512    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-cni-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-run-netns\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39cb27f9-a0fb-49bd-9a81-0a48771a9770-cni-binary-copy\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037632    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-run-netns\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-cni-multus\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037758    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-cni-multus\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-modprobe-d\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-kubernetes\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-run\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfzh\" (UniqueName: \"kubernetes.io/projected/6007e8ce-157f-403c-9c0d-651547e29295-kube-api-access-2mfzh\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-cni-netd\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.037993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.037988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-netns\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-cnibin\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-cni-binary-copy\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36dc0470-269b-4bed-9ab1-f1c0e63b2440-dbus\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2887a4ee-7198-42be-a636-3d8afd048849-etc-tuned\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2887a4ee-7198-42be-a636-3d8afd048849-tmp\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovn-node-metrics-cert\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r5nm\" (UniqueName: \"kubernetes.io/projected/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-kube-api-access-2r5nm\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-socket-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-systemd-units\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-kubelet\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-etc-kubernetes\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-system-cni-dir\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-os-release\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.038801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038320    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39cb27f9-a0fb-49bd-9a81-0a48771a9770-cni-binary-copy\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-sys-fs\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-var-lib-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038433    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-systemd-units\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-sys-fs\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038481    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-cni-netd\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038479    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-ovn\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038544    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-kubelet\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-cni-bin\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-cni-bin\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-systemd\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fsb\" (UniqueName: \"kubernetes.io/projected/2887a4ee-7198-42be-a636-3d8afd048849-kube-api-access-p2fsb\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-os-release\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e56b49-6747-48b3-891d-09f7547abfe4-hosts-file\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8t4k\" (UniqueName: \"kubernetes.io/projected/22e56b49-6747-48b3-891d-09f7547abfe4-kube-api-access-c8t4k\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.039674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysctl-conf\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-host\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-systemd\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038867    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-cni-bin\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovnkube-script-lib\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-serviceca\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ea25d04-2500-4c7b-b77d-b84c3ce92344-konnectivity-ca\") pod \"konnectivity-agent-ktlhf\" (UID: \"7ea25d04-2500-4c7b-b77d-b84c3ce92344\") " pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-cnibin\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038959    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.038991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-hostroot\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-node-log\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovnkube-config\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-env-overrides\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-socket-dir-parent\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-multus-certs\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zrk\" (UniqueName: \"kubernetes.io/projected/05388e7b-2246-4681-9c88-00caef6231de-kube-api-access-q9zrk\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.040158 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-var-lib-kubelet\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-registration-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-etc-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-socket-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-etc-kubernetes\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039292    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-netns\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039304    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-registration-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039320    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-var-lib-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039338    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039347    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-etc-openvswitch\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039372    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-ovn\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039375    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039399    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-node-log\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-daemon-config\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysctl-d\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2w2r\" (UniqueName: \"kubernetes.io/projected/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-kube-api-access-d2w2r\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.040866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-system-cni-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-conf-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039578    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-cnibin\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4jh\" (UniqueName: \"kubernetes.io/projected/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-kube-api-access-sl4jh\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-run-systemd\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6007e8ce-157f-403c-9c0d-651547e29295-host-slash\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-kubelet\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039670    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-system-cni-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-log-socket\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039703    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-hostroot\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxpvg\" (UniqueName: \"kubernetes.io/projected/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-kube-api-access-kxpvg\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ea25d04-2500-4c7b-b77d-b84c3ce92344-agent-certs\") pod \"konnectivity-agent-ktlhf\" (UID: \"7ea25d04-2500-4c7b-b77d-b84c3ce92344\") " pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039766    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-os-release\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36dc0470-269b-4bed-9ab1-f1c0e63b2440-kubelet-config\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-slash\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-k8s-cni-cncf-io\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-env-overrides\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039915    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-conf-dir\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.041334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039967    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-var-lib-cni-bin\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040004    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovnkube-config\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.039871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7lb\" (UniqueName: \"kubernetes.io/projected/39cb27f9-a0fb-49bd-9a81-0a48771a9770-kube-api-access-db7lb\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-kubelet\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040104    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-slash\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040357    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ea25d04-2500-4c7b-b77d-b84c3ce92344-konnectivity-ca\") pod \"konnectivity-agent-ktlhf\" (UID: \"7ea25d04-2500-4c7b-b77d-b84c3ce92344\") " pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040394    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-serviceca\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040439    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-multus-certs\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040454    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040487    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-host-run-k8s-cni-cncf-io\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040526    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-daemon-config\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39cb27f9-a0fb-49bd-9a81-0a48771a9770-multus-socket-dir-parent\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-log-socket\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.042144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.040799    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovnkube-script-lib\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.042795 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.042479    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-ovn-node-metrics-cert\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.042892 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.042870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ea25d04-2500-4c7b-b77d-b84c3ce92344-agent-certs\") pod \"konnectivity-agent-ktlhf\" (UID: \"7ea25d04-2500-4c7b-b77d-b84c3ce92344\") " pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:28:45.049899 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.049432    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 16:28:45.049899 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.049456    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 16:28:45.049899 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.049469    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v9x9g for pod openshift-network-diagnostics/network-check-target-skj5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:45.049899 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.049625    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g podName:1bef6b96-c17a-4f17-8767-62750be0051a nodeName:}" failed. No retries permitted until 2026-07-06 16:28:45.54960291 +0000 UTC m=+2.987113800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v9x9g" (UniqueName: "kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g") pod "network-check-target-skj5h" (UID: "1bef6b96-c17a-4f17-8767-62750be0051a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:45.051969 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.051942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r5nm\" (UniqueName: \"kubernetes.io/projected/f8de9897-da9a-4c99-bd4e-9bdfab9d037f-kube-api-access-2r5nm\") pod \"node-ca-smb7q\" (UID: \"f8de9897-da9a-4c99-bd4e-9bdfab9d037f\") " pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:45.052421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.052338    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2w2r\" (UniqueName: \"kubernetes.io/projected/b5999d48-f7c0-45e5-a3a6-5941634ba6ee-kube-api-access-d2w2r\") pod \"aws-ebs-csi-driver-node-75ktt\" (UID: \"b5999d48-f7c0-45e5-a3a6-5941634ba6ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.052421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.052398    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7lb\" (UniqueName: \"kubernetes.io/projected/39cb27f9-a0fb-49bd-9a81-0a48771a9770-kube-api-access-db7lb\") pod \"multus-49bs7\" (UID: \"39cb27f9-a0fb-49bd-9a81-0a48771a9770\") " pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.052850 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.052829    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxpvg\" (UniqueName: \"kubernetes.io/projected/c7c741dd-9fd7-4de1-ab91-bf4735ddd51d-kube-api-access-kxpvg\") pod \"ovnkube-node-j748n\" (UID: \"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.112743 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.112684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal" event={"ID":"971ca3d09392685f8bb4e145bd0a3e41","Type":"ContainerStarted","Data":"a0fa50acd020e5f836924176f6f310f1e93e67e90833d0c8c6c746f7e8745aa0"}
Jul 06 16:28:45.114614 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.114589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal" event={"ID":"9a9545f43e4880450f1a48b17d8b749c","Type":"ContainerStarted","Data":"92e0043c94b66fc7f592b5d0d49729ad782b79e69d74f44bad8e1e423d264c3a"}
Jul 06 16:28:45.140773 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-cni-binary-copy\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.140958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.140958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36dc0470-269b-4bed-9ab1-f1c0e63b2440-dbus\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.140958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2887a4ee-7198-42be-a636-3d8afd048849-etc-tuned\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.140958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2887a4ee-7198-42be-a636-3d8afd048849-tmp\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.140958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-system-cni-dir\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.140958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-os-release\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.140958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-systemd\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.140958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fsb\" (UniqueName: \"kubernetes.io/projected/2887a4ee-7198-42be-a636-3d8afd048849-kube-api-access-p2fsb\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.140979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e56b49-6747-48b3-891d-09f7547abfe4-hosts-file\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8t4k\" (UniqueName: \"kubernetes.io/projected/22e56b49-6747-48b3-891d-09f7547abfe4-kube-api-access-c8t4k\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysctl-conf\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-host\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141049    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zrk\" (UniqueName: \"kubernetes.io/projected/05388e7b-2246-4681-9c88-00caef6231de-kube-api-access-q9zrk\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141116    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-systemd\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-var-lib-kubelet\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysctl-d\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4jh\" (UniqueName: \"kubernetes.io/projected/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-kube-api-access-sl4jh\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6007e8ce-157f-403c-9c0d-651547e29295-host-slash\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36dc0470-269b-4bed-9ab1-f1c0e63b2440-kubelet-config\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e56b49-6747-48b3-891d-09f7547abfe4-tmp-dir\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysconfig\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6007e8ce-157f-403c-9c0d-651547e29295-iptables-alerter-script\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.141340 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-sys\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141339    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-cni-binary-copy\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-lib-modules\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-modprobe-d\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141441    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36dc0470-269b-4bed-9ab1-f1c0e63b2440-dbus\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-kubernetes\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-run\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfzh\" (UniqueName: \"kubernetes.io/projected/6007e8ce-157f-403c-9c0d-651547e29295-kube-api-access-2mfzh\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-cnibin\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141619    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-cnibin\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141655    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6007e8ce-157f-403c-9c0d-651547e29295-host-slash\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36dc0470-269b-4bed-9ab1-f1c0e63b2440-kubelet-config\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141767    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22e56b49-6747-48b3-891d-09f7547abfe4-hosts-file\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.141971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.141894    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e56b49-6747-48b3-891d-09f7547abfe4-tmp-dir\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.142393 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysctl-conf\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142393 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142108    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-host\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142393 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysconfig\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142393 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.142294    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:45.142393 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.142340    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret podName:36dc0470-269b-4bed-9ab1-f1c0e63b2440 nodeName:}" failed. No retries permitted until 2026-07-06 16:28:45.642327146 +0000 UTC m=+3.079838032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret") pod "global-pull-secret-syncer-slpn6" (UID: "36dc0470-269b-4bed-9ab1-f1c0e63b2440") : object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:45.142587 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6007e8ce-157f-403c-9c0d-651547e29295-iptables-alerter-script\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.142587 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142469    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-var-lib-kubelet\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142587 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142544    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-sys\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142724 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142588    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-sysctl-d\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142724 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-kubernetes\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142724 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.142680    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:45.142724 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142682    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-etc-modprobe-d\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142724 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.142722    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:28:45.642707863 +0000 UTC m=+3.080218750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:45.142981 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142771    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-run\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142981 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142673    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2887a4ee-7198-42be-a636-3d8afd048849-lib-modules\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.142981 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-system-cni-dir\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.142981 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.142960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05388e7b-2246-4681-9c88-00caef6231de-os-release\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.143176 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.143161    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.143648 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.143626    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2887a4ee-7198-42be-a636-3d8afd048849-etc-tuned\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.143748 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.143688    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05388e7b-2246-4681-9c88-00caef6231de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.145172 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.145137    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2887a4ee-7198-42be-a636-3d8afd048849-tmp\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.155045 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.154977    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4jh\" (UniqueName: \"kubernetes.io/projected/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-kube-api-access-sl4jh\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:45.156197 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.156171    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zrk\" (UniqueName: \"kubernetes.io/projected/05388e7b-2246-4681-9c88-00caef6231de-kube-api-access-q9zrk\") pod \"multus-additional-cni-plugins-7p6kj\" (UID: \"05388e7b-2246-4681-9c88-00caef6231de\") " pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.157317 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.157292    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfzh\" (UniqueName: \"kubernetes.io/projected/6007e8ce-157f-403c-9c0d-651547e29295-kube-api-access-2mfzh\") pod \"iptables-alerter-czt6b\" (UID: \"6007e8ce-157f-403c-9c0d-651547e29295\") " pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.157959 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.157937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fsb\" (UniqueName: \"kubernetes.io/projected/2887a4ee-7198-42be-a636-3d8afd048849-kube-api-access-p2fsb\") pod \"tuned-phxxj\" (UID: \"2887a4ee-7198-42be-a636-3d8afd048849\") " pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.159296 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.159269    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8t4k\" (UniqueName: \"kubernetes.io/projected/22e56b49-6747-48b3-891d-09f7547abfe4-kube-api-access-c8t4k\") pod \"node-resolver-8k9x7\" (UID: \"22e56b49-6747-48b3-891d-09f7547abfe4\") " pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.231848 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.231811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt"
Jul 06 16:28:45.240687 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.240660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:28:45.243046 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:45.243015    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5999d48_f7c0_45e5_a3a6_5941634ba6ee.slice/crio-cf7a53de1c43b1e91d090d6fc0586a549f5640aab0994cb9aef6f391debd94e7 WatchSource:0}: Error finding container cf7a53de1c43b1e91d090d6fc0586a549f5640aab0994cb9aef6f391debd94e7: Status 404 returned error can't find the container with id cf7a53de1c43b1e91d090d6fc0586a549f5640aab0994cb9aef6f391debd94e7
Jul 06 16:28:45.243172 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.243133    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 16:28:45.249571 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:45.249546    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c741dd_9fd7_4de1_ab91_bf4735ddd51d.slice/crio-b92ed68ae0f64144028d6ffa9508b0d7b6f6f9efa5494df87c441e06f4e53552 WatchSource:0}: Error finding container b92ed68ae0f64144028d6ffa9508b0d7b6f6f9efa5494df87c441e06f4e53552: Status 404 returned error can't find the container with id b92ed68ae0f64144028d6ffa9508b0d7b6f6f9efa5494df87c441e06f4e53552
Jul 06 16:28:45.254657 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.254638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-smb7q"
Jul 06 16:28:45.259638 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.259613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:28:45.269620 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.269386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49bs7"
Jul 06 16:28:45.278029 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.277997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-czt6b"
Jul 06 16:28:45.286097 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.286067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8k9x7"
Jul 06 16:28:45.293153 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.293127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-phxxj"
Jul 06 16:28:45.301357 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.301334    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7p6kj"
Jul 06 16:28:45.402054 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.402021    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 16:28:45.646352 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.646321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:45.646485 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.646368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:45.646485 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.646405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:45.646599 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.646527    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 16:28:45.646599 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.646546    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 16:28:45.646599 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.646557    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v9x9g for pod openshift-network-diagnostics/network-check-target-skj5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:45.646720 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.646601    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g podName:1bef6b96-c17a-4f17-8767-62750be0051a nodeName:}" failed. No retries permitted until 2026-07-06 16:28:46.646586777 +0000 UTC m=+4.084097664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9x9g" (UniqueName: "kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g") pod "network-check-target-skj5h" (UID: "1bef6b96-c17a-4f17-8767-62750be0051a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:45.646910 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.646896    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:45.646957 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.646944    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret podName:36dc0470-269b-4bed-9ab1-f1c0e63b2440 nodeName:}" failed. No retries permitted until 2026-07-06 16:28:46.646930883 +0000 UTC m=+4.084441770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret") pod "global-pull-secret-syncer-slpn6" (UID: "36dc0470-269b-4bed-9ab1-f1c0e63b2440") : object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:45.646999 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.646990    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:45.647028 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:45.647014    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:28:46.647007446 +0000 UTC m=+4.084518332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:45.979019 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.978919    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 16:23:44 +0000 UTC" deadline="2028-04-05 20:48:12.54462229 +0000 UTC"
Jul 06 16:28:45.979019 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:45.978970    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15340h19m26.565657885s"
Jul 06 16:28:46.117972 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:46.117937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt" event={"ID":"b5999d48-f7c0-45e5-a3a6-5941634ba6ee","Type":"ContainerStarted","Data":"cf7a53de1c43b1e91d090d6fc0586a549f5640aab0994cb9aef6f391debd94e7"}
Jul 06 16:28:46.119119 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:46.119091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"b92ed68ae0f64144028d6ffa9508b0d7b6f6f9efa5494df87c441e06f4e53552"}
Jul 06 16:28:46.290776 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:46.290599    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8de9897_da9a_4c99_bd4e_9bdfab9d037f.slice/crio-f60710083884a3cf8bfe32f508672213ae481130986ca4cfeb03ad559ce935d7 WatchSource:0}: Error finding container f60710083884a3cf8bfe32f508672213ae481130986ca4cfeb03ad559ce935d7: Status 404 returned error can't find the container with id f60710083884a3cf8bfe32f508672213ae481130986ca4cfeb03ad559ce935d7
Jul 06 16:28:46.290971 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:46.290955    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea25d04_2500_4c7b_b77d_b84c3ce92344.slice/crio-4f36db36b87a051964667292dda02b92ee42c71d92c2944db8f429f71eb0a4e1 WatchSource:0}: Error finding container 4f36db36b87a051964667292dda02b92ee42c71d92c2944db8f429f71eb0a4e1: Status 404 returned error can't find the container with id 4f36db36b87a051964667292dda02b92ee42c71d92c2944db8f429f71eb0a4e1
Jul 06 16:28:46.292094 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:46.292062    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2887a4ee_7198_42be_a636_3d8afd048849.slice/crio-e220a85a497af2056bcc0e24f88ab169843c9c8effbc9748890bc8b15669e9ff WatchSource:0}: Error finding container e220a85a497af2056bcc0e24f88ab169843c9c8effbc9748890bc8b15669e9ff: Status 404 returned error can't find the container with id e220a85a497af2056bcc0e24f88ab169843c9c8effbc9748890bc8b15669e9ff
Jul 06 16:28:46.294612 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:46.294591    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cb27f9_a0fb_49bd_9a81_0a48771a9770.slice/crio-7e90e1c66bc03170f830abab9d2a5d46109197330049ddea4cab60f563a1e126 WatchSource:0}: Error finding container 7e90e1c66bc03170f830abab9d2a5d46109197330049ddea4cab60f563a1e126: Status 404 returned error can't find the container with id 7e90e1c66bc03170f830abab9d2a5d46109197330049ddea4cab60f563a1e126
Jul 06 16:28:46.302067 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:46.298848    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05388e7b_2246_4681_9c88_00caef6231de.slice/crio-d289d11c161e3d875a5b616a0679b96d24d300245e0b678ae83058ab387b657f WatchSource:0}: Error finding container d289d11c161e3d875a5b616a0679b96d24d300245e0b678ae83058ab387b657f: Status 404 returned error can't find the container with id d289d11c161e3d875a5b616a0679b96d24d300245e0b678ae83058ab387b657f
Jul 06 16:28:46.302067 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:28:46.300162    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e56b49_6747_48b3_891d_09f7547abfe4.slice/crio-5e5ddd80696932f1555d3a32821d94045c3ce6a0de1513a3658a24820e3eeca2 WatchSource:0}: Error finding container 5e5ddd80696932f1555d3a32821d94045c3ce6a0de1513a3658a24820e3eeca2: Status 404 returned error can't find the container with id 5e5ddd80696932f1555d3a32821d94045c3ce6a0de1513a3658a24820e3eeca2
Jul 06 16:28:46.654965 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:46.654936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:46.655071 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:46.655001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:46.655186 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:46.655159    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:46.655186 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:46.655165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:46.655272 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:46.655231    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:28:48.655211832 +0000 UTC m=+6.092722724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:46.655320 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:46.655302    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:46.655371 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:46.655324    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 16:28:46.655371 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:46.655339    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 16:28:46.655371 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:46.655352    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v9x9g for pod openshift-network-diagnostics/network-check-target-skj5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:46.655488 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:46.655352    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret podName:36dc0470-269b-4bed-9ab1-f1c0e63b2440 nodeName:}" failed. No retries permitted until 2026-07-06 16:28:48.655336254 +0000 UTC m=+6.092847156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret") pod "global-pull-secret-syncer-slpn6" (UID: "36dc0470-269b-4bed-9ab1-f1c0e63b2440") : object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:46.655488 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:46.655394    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g podName:1bef6b96-c17a-4f17-8767-62750be0051a nodeName:}" failed. No retries permitted until 2026-07-06 16:28:48.655379831 +0000 UTC m=+6.092890722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9x9g" (UniqueName: "kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g") pod "network-check-target-skj5h" (UID: "1bef6b96-c17a-4f17-8767-62750be0051a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:46.979388 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:46.979244    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 16:23:44 +0000 UTC" deadline="2027-12-07 08:09:23.00110325 +0000 UTC"
Jul 06 16:28:46.979388 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:46.979307    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12447h40m36.02180057s"
Jul 06 16:28:47.109380 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.108658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:47.109380 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:47.108806    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:28:47.109380 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.109219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:47.109380 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:47.109312    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:28:47.109935 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.109799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:47.109935 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:47.109899    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:28:47.124747 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.124667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8k9x7" event={"ID":"22e56b49-6747-48b3-891d-09f7547abfe4","Type":"ContainerStarted","Data":"5e5ddd80696932f1555d3a32821d94045c3ce6a0de1513a3658a24820e3eeca2"}
Jul 06 16:28:47.128397 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.128362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-phxxj" event={"ID":"2887a4ee-7198-42be-a636-3d8afd048849","Type":"ContainerStarted","Data":"e220a85a497af2056bcc0e24f88ab169843c9c8effbc9748890bc8b15669e9ff"}
Jul 06 16:28:47.130822 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.130789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktlhf" event={"ID":"7ea25d04-2500-4c7b-b77d-b84c3ce92344","Type":"ContainerStarted","Data":"4f36db36b87a051964667292dda02b92ee42c71d92c2944db8f429f71eb0a4e1"}
Jul 06 16:28:47.132214 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.132187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-smb7q" event={"ID":"f8de9897-da9a-4c99-bd4e-9bdfab9d037f","Type":"ContainerStarted","Data":"f60710083884a3cf8bfe32f508672213ae481130986ca4cfeb03ad559ce935d7"}
Jul 06 16:28:47.134770 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.134743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal" event={"ID":"9a9545f43e4880450f1a48b17d8b749c","Type":"ContainerStarted","Data":"5ccdf142bf7ed56789dc71f29787608121b506953ee43196aaeda04aa0e1f1be"}
Jul 06 16:28:47.136998 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.136631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerStarted","Data":"d289d11c161e3d875a5b616a0679b96d24d300245e0b678ae83058ab387b657f"}
Jul 06 16:28:47.138159 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.138099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-czt6b" event={"ID":"6007e8ce-157f-403c-9c0d-651547e29295","Type":"ContainerStarted","Data":"06885c9ec77b151c2fc5dc990de5bf73c3c57dc6297e8bad00f72a15bd50a19b"}
Jul 06 16:28:47.140187 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.140134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49bs7" event={"ID":"39cb27f9-a0fb-49bd-9a81-0a48771a9770","Type":"ContainerStarted","Data":"7e90e1c66bc03170f830abab9d2a5d46109197330049ddea4cab60f563a1e126"}
Jul 06 16:28:47.143212 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.142564    2578 generic.go:356] "Generic (PLEG): container finished" podID="971ca3d09392685f8bb4e145bd0a3e41" containerID="90879b75bd9e2b4826a4a0faa5a516f790c03bfae6b76991a20f8f89a054cbb8" exitCode=0
Jul 06 16:28:47.143212 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.142605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal" event={"ID":"971ca3d09392685f8bb4e145bd0a3e41","Type":"ContainerDied","Data":"90879b75bd9e2b4826a4a0faa5a516f790c03bfae6b76991a20f8f89a054cbb8"}
Jul 06 16:28:47.168198 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:47.167607    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-118.ec2.internal" podStartSLOduration=3.167585371 podStartE2EDuration="3.167585371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:47.150604531 +0000 UTC m=+4.588115456" watchObservedRunningTime="2026-07-06 16:28:47.167585371 +0000 UTC m=+4.605096283"
Jul 06 16:28:48.164627 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:48.164578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal" event={"ID":"971ca3d09392685f8bb4e145bd0a3e41","Type":"ContainerStarted","Data":"6a968bf3fa400ba8696e00af664a5bf73289b3def284ba6e754e04ad96bf1e6f"}
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:48.673745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:48.673813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:48.673855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:48.673988    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:48.674006    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:48.674020    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v9x9g for pod openshift-network-diagnostics/network-check-target-skj5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:48.674080    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g podName:1bef6b96-c17a-4f17-8767-62750be0051a nodeName:}" failed. No retries permitted until 2026-07-06 16:28:52.674061803 +0000 UTC m=+10.111572693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9x9g" (UniqueName: "kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g") pod "network-check-target-skj5h" (UID: "1bef6b96-c17a-4f17-8767-62750be0051a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:48.674495    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:48.674563    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret podName:36dc0470-269b-4bed-9ab1-f1c0e63b2440 nodeName:}" failed. No retries permitted until 2026-07-06 16:28:52.67454664 +0000 UTC m=+10.112057554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret") pod "global-pull-secret-syncer-slpn6" (UID: "36dc0470-269b-4bed-9ab1-f1c0e63b2440") : object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:48.674633    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:48.674698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:48.674667    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:28:52.674655045 +0000 UTC m=+10.112165945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:49.109443 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:49.109386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:49.109639 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:49.109532    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:28:49.109639 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:49.109592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:49.109729 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:49.109665    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:28:49.109729 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:49.109709    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:49.109833 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:49.109795    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:28:51.107664 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:51.107629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:51.107664 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:51.107676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:51.108238 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:51.107772    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:28:51.108238 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:51.108199    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:28:51.108348 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:51.108311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:51.108436 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:51.108407    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:28:52.710621 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:52.710582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:52.711077 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:52.710642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:52.711077 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:52.710669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:52.711077 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:52.710773    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:52.711077 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:52.710820    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret podName:36dc0470-269b-4bed-9ab1-f1c0e63b2440 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:00.710807119 +0000 UTC m=+18.148318009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret") pod "global-pull-secret-syncer-slpn6" (UID: "36dc0470-269b-4bed-9ab1-f1c0e63b2440") : object "kube-system"/"original-pull-secret" not registered
Jul 06 16:28:52.711294 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:52.711141    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:52.711294 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:52.711183    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:29:00.711167277 +0000 UTC m=+18.148678179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:28:52.711294 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:52.711241    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 16:28:52.711294 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:52.711253    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 16:28:52.711294 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:52.711267    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v9x9g for pod openshift-network-diagnostics/network-check-target-skj5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:52.711294 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:52.711292    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g podName:1bef6b96-c17a-4f17-8767-62750be0051a nodeName:}" failed. No retries permitted until 2026-07-06 16:29:00.71128536 +0000 UTC m=+18.148796249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9x9g" (UniqueName: "kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g") pod "network-check-target-skj5h" (UID: "1bef6b96-c17a-4f17-8767-62750be0051a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:28:53.108204 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:53.108168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:53.108387 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:53.108213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:53.108387 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:53.108296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:53.108387 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:53.108339    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:28:53.108536 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:53.108390    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:28:53.108536 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:53.108406    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:28:55.108649 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:55.108619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:55.109131 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:55.108732    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:28:55.109131 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:55.109064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:55.109131 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:55.109124    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:28:55.109283 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:55.109169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:55.109283 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:55.109215    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:28:57.107597 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.107043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:57.107597 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.107105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:57.107597 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:57.107299    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:28:57.107597 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:57.107363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:28:57.107597 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.107138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:57.107597 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:57.107424    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:28:57.187746 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.187667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerStarted","Data":"eff9457382744c88d940e7f917756839d84c74204a2ce7f9ae1078ca058e7aaa"}
Jul 06 16:28:57.189465 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.189432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt" event={"ID":"b5999d48-f7c0-45e5-a3a6-5941634ba6ee","Type":"ContainerStarted","Data":"8496b5fb3ec63555c88eacf8cf4c8be1e4106b7a11196fc88b05103107fd60f5"}
Jul 06 16:28:57.191108 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.191079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8k9x7" event={"ID":"22e56b49-6747-48b3-891d-09f7547abfe4","Type":"ContainerStarted","Data":"5ac85a69e1cf27c48f1e4ea3f54658adb46c8e9eef9c762cbee9aadafe63350d"}
Jul 06 16:28:57.192789 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.192766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-phxxj" event={"ID":"2887a4ee-7198-42be-a636-3d8afd048849","Type":"ContainerStarted","Data":"2041cdb4890ac6931759ccbd41228e48bcefeac28fddd95bd50edfc4f5d4f0fd"}
Jul 06 16:28:57.194745 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.194721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktlhf" event={"ID":"7ea25d04-2500-4c7b-b77d-b84c3ce92344","Type":"ContainerStarted","Data":"6e631b1828e7cdfd91578cba4542f4d85a819b04298b208e2f185818f05f7331"}
Jul 06 16:28:57.197743 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.196295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-smb7q" event={"ID":"f8de9897-da9a-4c99-bd4e-9bdfab9d037f","Type":"ContainerStarted","Data":"1591f23a3f90620ab5371b55990a1aed38e7a5bd413edb0147b4b524b5902c25"}
Jul 06 16:28:57.212921 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.212856    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-118.ec2.internal" podStartSLOduration=13.212810817 podStartE2EDuration="13.212810817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:48.180066039 +0000 UTC m=+5.617576982" watchObservedRunningTime="2026-07-06 16:28:57.212810817 +0000 UTC m=+14.650321728"
Jul 06 16:28:57.244933 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.244886    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-phxxj" podStartSLOduration=3.587352254 podStartE2EDuration="14.244868533s" totalImagesPullingTime="10.657516279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:57.22996614 +0000 UTC m=+14.667477076" watchObservedRunningTime="2026-07-06 16:28:57.244868533 +0000 UTC m=+14.682379493"
Jul 06 16:28:57.245049 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.245015    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-smb7q" podStartSLOduration=3.793166771 podStartE2EDuration="14.245009916s" totalImagesPullingTime="10.451843145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:57.244339759 +0000 UTC m=+14.681850669" watchObservedRunningTime="2026-07-06 16:28:57.245009916 +0000 UTC m=+14.682520825"
Jul 06 16:28:57.259979 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.259924    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ktlhf" podStartSLOduration=3.810528227 podStartE2EDuration="14.259904524s" totalImagesPullingTime="10.449376297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:57.259376471 +0000 UTC m=+14.696887407" watchObservedRunningTime="2026-07-06 16:28:57.259904524 +0000 UTC m=+14.697415435"
Jul 06 16:28:57.275270 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:57.275212    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8k9x7" podStartSLOduration=3.836496146 podStartE2EDuration="14.27518864s" totalImagesPullingTime="10.438692494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:57.274594863 +0000 UTC m=+14.712105773" watchObservedRunningTime="2026-07-06 16:28:57.27518864 +0000 UTC m=+14.712699551"
Jul 06 16:28:59.106455 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:59.106420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:28:59.107075 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:59.106590    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:28:59.107075 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:59.107010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:28:59.107174 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:59.107099    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:28:59.107223 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:59.107187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:28:59.107285 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:28:59.107264    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:28:59.207039 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:59.206996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-czt6b" event={"ID":"6007e8ce-157f-403c-9c0d-651547e29295","Type":"ContainerStarted","Data":"1ed96e42734fd60d927154b078b8d4a8834acbfc87ab781a729ea5f5edebd2e2"}
Jul 06 16:28:59.222699 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:59.222631    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-czt6b" podStartSLOduration=5.778024564 podStartE2EDuration="16.222610945s" totalImagesPullingTime="10.444586381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:59.221910182 +0000 UTC m=+16.659421110" watchObservedRunningTime="2026-07-06 16:28:59.222610945 +0000 UTC m=+16.660121854"
Jul 06 16:28:59.364687 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:59.364597    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlhf_7ea25d04-2500-4c7b-b77d-b84c3ce92344/konnectivity-agent/0.log" limit=1024
Jul 06 16:28:59.422212 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:28:59.422182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlhf_7ea25d04-2500-4c7b-b77d-b84c3ce92344/konnectivity-agent/0.log" limit=1024
Jul 06 16:29:00.781459 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:00.781419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:00.781459 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:00.781461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:00.781557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:00.781690    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:00.781709    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:00.781719    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v9x9g for pod openshift-network-diagnostics/network-check-target-skj5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:00.781690    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:00.781774    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g podName:1bef6b96-c17a-4f17-8767-62750be0051a nodeName:}" failed. No retries permitted until 2026-07-06 16:29:16.781754884 +0000 UTC m=+34.219265838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9x9g" (UniqueName: "kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g") pod "network-check-target-skj5h" (UID: "1bef6b96-c17a-4f17-8767-62750be0051a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:00.781801    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:00.781836    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:29:16.781814935 +0000 UTC m=+34.219325821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:29:00.782137 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:00.781860    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret podName:36dc0470-269b-4bed-9ab1-f1c0e63b2440 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:16.781844098 +0000 UTC m=+34.219355004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret") pod "global-pull-secret-syncer-slpn6" (UID: "36dc0470-269b-4bed-9ab1-f1c0e63b2440") : object "kube-system"/"original-pull-secret" not registered
Jul 06 16:29:01.106692 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:01.106653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:01.106692 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:01.106684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:01.106901 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:01.106763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:01.106901 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:01.106775    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:01.106901 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:01.106841    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:01.107062 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:01.106976    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:01.213751 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:01.213714    2578 generic.go:356] "Generic (PLEG): container finished" podID="05388e7b-2246-4681-9c88-00caef6231de" containerID="eff9457382744c88d940e7f917756839d84c74204a2ce7f9ae1078ca058e7aaa" exitCode=0
Jul 06 16:29:01.213906 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:01.213773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerDied","Data":"eff9457382744c88d940e7f917756839d84c74204a2ce7f9ae1078ca058e7aaa"}
Jul 06 16:29:01.759954 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:01.759916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlhf_7ea25d04-2500-4c7b-b77d-b84c3ce92344/konnectivity-agent/0.log" limit=1024
Jul 06 16:29:01.854624 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:01.854588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlhf_7ea25d04-2500-4c7b-b77d-b84c3ce92344/konnectivity-agent/0.log" limit=1024
Jul 06 16:29:02.009568 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:02.009536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:29:02.010220 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:02.010150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:29:02.048027 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:02.047997    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:29:02.048610 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:02.048589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ktlhf"
Jul 06 16:29:02.107923 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:02.107884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlhf_7ea25d04-2500-4c7b-b77d-b84c3ce92344/konnectivity-agent/0.log" limit=1024
Jul 06 16:29:03.107839 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:03.107637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:03.108273 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:03.107658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:03.108273 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:03.107917    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:03.108273 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:03.107684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:03.108273 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:03.108020    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:03.108273 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:03.108080    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:05.107651 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:05.107395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:05.107651 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:05.107572    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:05.108199 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:05.108022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:05.108199 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:05.108126    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:05.108279 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:05.108265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:05.108393 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:05.108374    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:05.566361 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:05.566074    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 16:29:06.004178 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.004066    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T16:29:05.566325618Z","UUID":"38319db3-891c-4df4-a7e8-2608eb4ef9f6","Handler":null,"Name":"","Endpoint":""}
Jul 06 16:29:06.006251 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.006221    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 16:29:06.006407 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.006269    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 16:29:06.226978 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.226889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt" event={"ID":"b5999d48-f7c0-45e5-a3a6-5941634ba6ee","Type":"ContainerStarted","Data":"7d1a39a7114d7e78fe8ad86d47f33ff79a2cf12b41636d7e2dbc4ea88e2df1a3"}
Jul 06 16:29:06.231143 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.231111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"3451d1e1dd6a91d6f0877c2e67702a71ce633971d4e270355d0a9632711ea059"}
Jul 06 16:29:06.231298 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.231197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"0183e0e55906a89171a0b6b9b20f017e479bee8c4dd17f92785dd721d1241b89"}
Jul 06 16:29:06.231298 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.231246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"2e76ebd078553b5fe59126be682b682e43a0204d1889280d97dc2bcddd06486f"}
Jul 06 16:29:06.231298 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.231281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"fd3abc501f7e89b4547af9d7bf39e300f46fc2a097f51887b120551228ebe420"}
Jul 06 16:29:06.231463 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.231307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"40b0bf2ffc00bade5fe178eb1a1479ff77dd5a0bdd235aab7fa4efd95383fae9"}
Jul 06 16:29:06.231463 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.231333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"897b85b65f31a97dbc2489f74a65b642dfbb2d0101e78a30e805fb91bb7d6b32"}
Jul 06 16:29:06.233669 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:06.233616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49bs7" event={"ID":"39cb27f9-a0fb-49bd-9a81-0a48771a9770","Type":"ContainerStarted","Data":"8d3bbea2a33eb01e57429ed97ec639605c7e6ae1af8821fcff60b295afe9ab09"}
Jul 06 16:29:07.106989 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:07.106961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:07.107141 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:07.107079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:07.107141 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:07.107094    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:07.107141 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:07.107114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:07.107292 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:07.107193    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:07.107342 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:07.107285    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:07.238454 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:07.238372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt" event={"ID":"b5999d48-f7c0-45e5-a3a6-5941634ba6ee","Type":"ContainerStarted","Data":"4b5c6d02409cf65904b7b293013fe3493f21fa621fab3927a50e103d60a9e7e8"}
Jul 06 16:29:07.258672 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:07.258608    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-49bs7" podStartSLOduration=5.263302445 podStartE2EDuration="24.258585325s" totalImagesPullingTime="18.99528288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:06.252595187 +0000 UTC m=+23.690106123" watchObservedRunningTime="2026-07-06 16:29:07.258585325 +0000 UTC m=+24.696096235"
Jul 06 16:29:07.259065 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:07.259033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-75ktt" podStartSLOduration=2.929084283 podStartE2EDuration="24.259021627s" totalImagesPullingTime="21.329937344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:07.257462697 +0000 UTC m=+24.694973626" watchObservedRunningTime="2026-07-06 16:29:07.259021627 +0000 UTC m=+24.696532536"
Jul 06 16:29:09.107051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:09.107013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:09.107473 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:09.107013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:09.107473 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:09.107132    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:09.107473 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:09.107151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:09.107473 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:09.107240    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:09.107473 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:09.107374    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:10.248478 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:10.248446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"f038a3a6bbca6ede05eddaf3ca6ba8a6e34f12d961c7638a04acbf360bea5eb7"}
Jul 06 16:29:10.250157 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:10.250137    2578 generic.go:356] "Generic (PLEG): container finished" podID="05388e7b-2246-4681-9c88-00caef6231de" containerID="dedf510dd87494b62303e1397866148544346be18e624799e3cf3ab023bd1b9c" exitCode=0
Jul 06 16:29:10.250262 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:10.250183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerDied","Data":"dedf510dd87494b62303e1397866148544346be18e624799e3cf3ab023bd1b9c"}
Jul 06 16:29:11.106979 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:11.106878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:11.106979 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:11.106878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:11.107249 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:11.107006    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:11.107249 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:11.107072    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:11.107249 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:11.106887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:11.107249 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:11.107156    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:12.258435 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:12.258146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" event={"ID":"c7c741dd-9fd7-4de1-ab91-bf4735ddd51d","Type":"ContainerStarted","Data":"a3ca87bb9bf346e1967e72e4b253bf8353afa2ed72eb66d610d5b2ca54112863"}
Jul 06 16:29:12.258999 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:12.258474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:29:12.260771 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:12.260746    2578 generic.go:356] "Generic (PLEG): container finished" podID="05388e7b-2246-4681-9c88-00caef6231de" containerID="37029752301d935a51728da509168528b9aea034b4cb451a891f16b17780892e" exitCode=0
Jul 06 16:29:12.260908 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:12.260803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerDied","Data":"37029752301d935a51728da509168528b9aea034b4cb451a891f16b17780892e"}
Jul 06 16:29:12.274954 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:12.274926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:29:12.287945 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:12.287897    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j748n" podStartSLOduration=9.246471268 podStartE2EDuration="29.287883478s" totalImagesPullingTime="20.04141221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:12.28767936 +0000 UTC m=+29.725190284" watchObservedRunningTime="2026-07-06 16:29:12.287883478 +0000 UTC m=+29.725394386"
Jul 06 16:29:13.110974 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.110949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:13.111193 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.111015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:13.111193 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:13.111027    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:13.111193 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:13.111092    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:13.111193 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.111125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:13.111376 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:13.111228    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:13.263903 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.263605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:29:13.263903 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.263678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:29:13.279848 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.279823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:29:13.873838 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.873792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-slpn6"]
Jul 06 16:29:13.874016 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.873925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:13.874073 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:13.874039    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:13.874616 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.874590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-692bf"]
Jul 06 16:29:13.874721 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.874689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:13.874809 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:13.874789    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:13.887208 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.887177    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-skj5h"]
Jul 06 16:29:13.887380 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:13.887305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:13.887447 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:13.887409    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:14.268008 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:14.267819    2578 generic.go:356] "Generic (PLEG): container finished" podID="05388e7b-2246-4681-9c88-00caef6231de" containerID="a465ddf18cf40c26c5e0970c31b725eb01de53edb182a435dfd502730029e7be" exitCode=0
Jul 06 16:29:14.268008 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:14.267892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerDied","Data":"a465ddf18cf40c26c5e0970c31b725eb01de53edb182a435dfd502730029e7be"}
Jul 06 16:29:16.107463 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:16.107429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:16.107463 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:16.107452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:16.107998 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:16.107452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:16.107998 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.107597    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:29:16.107998 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.107735    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slpn6" podUID="36dc0470-269b-4bed-9ab1-f1c0e63b2440"
Jul 06 16:29:16.107998 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.107829    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-skj5h" podUID="1bef6b96-c17a-4f17-8767-62750be0051a"
Jul 06 16:29:16.796203 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:16.796171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:16.796203 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:16.796210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:16.796414 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:16.796243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:16.796414 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.796341    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:29:16.796414 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.796345    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 16:29:16.796414 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.796361    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 16:29:16.796414 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.796378    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 16:29:16.796414 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.796390    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v9x9g for pod openshift-network-diagnostics/network-check-target-skj5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:29:16.796414 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.796399    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:29:48.796383369 +0000 UTC m=+66.233894258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 16:29:16.796414 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.796412    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret podName:36dc0470-269b-4bed-9ab1-f1c0e63b2440 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:48.7964064 +0000 UTC m=+66.233917287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret") pod "global-pull-secret-syncer-slpn6" (UID: "36dc0470-269b-4bed-9ab1-f1c0e63b2440") : object "kube-system"/"original-pull-secret" not registered
Jul 06 16:29:16.796671 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:16.796430    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g podName:1bef6b96-c17a-4f17-8767-62750be0051a nodeName:}" failed. No retries permitted until 2026-07-06 16:29:48.796417037 +0000 UTC m=+66.233927923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9x9g" (UniqueName: "kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g") pod "network-check-target-skj5h" (UID: "1bef6b96-c17a-4f17-8767-62750be0051a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 16:29:17.864197 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.864168    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-118.ec2.internal" event="NodeReady"
Jul 06 16:29:17.864771 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.864312    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 16:29:17.908019 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.907981    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2nbjn"]
Jul 06 16:29:17.910874 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.910850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:17.911368 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.911349    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vgzzv"]
Jul 06 16:29:17.913140 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.913122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vgzzv"
Jul 06 16:29:17.913743 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.913607    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 16:29:17.913743 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.913609    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6rxcp\""
Jul 06 16:29:17.913743 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.913611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 16:29:17.915737 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.915606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 16:29:17.915906 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.915890    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 16:29:17.915980 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.915915    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 16:29:17.916067 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.916038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tbv5p\""
Jul 06 16:29:17.923938 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.923913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2nbjn"]
Jul 06 16:29:17.924054 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:17.923977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vgzzv"]
Jul 06 16:29:18.005331 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.005243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwdq\" (UniqueName: \"kubernetes.io/projected/685eb0d5-e105-422a-96fd-ff597ea88343-kube-api-access-mlwdq\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.005331 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.005306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tr9q\" (UniqueName: \"kubernetes.io/projected/3f60b419-0fe0-4291-a2be-eea12b95a646-kube-api-access-8tr9q\") pod \"ingress-canary-vgzzv\" (UID: \"3f60b419-0fe0-4291-a2be-eea12b95a646\") " pod="openshift-ingress-canary/ingress-canary-vgzzv"
Jul 06 16:29:18.005583 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.005395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685eb0d5-e105-422a-96fd-ff597ea88343-tmp-dir\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.005583 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.005456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.005583 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.005545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/685eb0d5-e105-422a-96fd-ff597ea88343-config-volume\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.005583 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.005570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f60b419-0fe0-4291-a2be-eea12b95a646-cert\") pod \"ingress-canary-vgzzv\" (UID: \"3f60b419-0fe0-4291-a2be-eea12b95a646\") " pod="openshift-ingress-canary/ingress-canary-vgzzv"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.106675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.106696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685eb0d5-e105-422a-96fd-ff597ea88343-tmp-dir\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.106703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.106753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:18.106840    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.106844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/685eb0d5-e105-422a-96fd-ff597ea88343-config-volume\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.106873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f60b419-0fe0-4291-a2be-eea12b95a646-cert\") pod \"ingress-canary-vgzzv\" (UID: \"3f60b419-0fe0-4291-a2be-eea12b95a646\") " pod="openshift-ingress-canary/ingress-canary-vgzzv"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.106873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:18.106891    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:18.606877798 +0000 UTC m=+36.044388685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.106969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwdq\" (UniqueName: \"kubernetes.io/projected/685eb0d5-e105-422a-96fd-ff597ea88343-kube-api-access-mlwdq\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.107004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tr9q\" (UniqueName: \"kubernetes.io/projected/3f60b419-0fe0-4291-a2be-eea12b95a646-kube-api-access-8tr9q\") pod \"ingress-canary-vgzzv\" (UID: \"3f60b419-0fe0-4291-a2be-eea12b95a646\") " pod="openshift-ingress-canary/ingress-canary-vgzzv"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.107060    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685eb0d5-e105-422a-96fd-ff597ea88343-tmp-dir\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.107537 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.107380    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/685eb0d5-e105-422a-96fd-ff597ea88343-config-volume\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.110462 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.110432    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 16:29:18.110462 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.110447    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 16:29:18.110702 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.110675    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jhwls\""
Jul 06 16:29:18.111566 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.110767    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jqc2r\""
Jul 06 16:29:18.111566 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.110849    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 16:29:18.111566 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.111455    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 16:29:18.112876 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.112843    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f60b419-0fe0-4291-a2be-eea12b95a646-cert\") pod \"ingress-canary-vgzzv\" (UID: \"3f60b419-0fe0-4291-a2be-eea12b95a646\") " pod="openshift-ingress-canary/ingress-canary-vgzzv"
Jul 06 16:29:18.119066 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.119038    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwdq\" (UniqueName: \"kubernetes.io/projected/685eb0d5-e105-422a-96fd-ff597ea88343-kube-api-access-mlwdq\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.119206 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.119175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tr9q\" (UniqueName: \"kubernetes.io/projected/3f60b419-0fe0-4291-a2be-eea12b95a646-kube-api-access-8tr9q\") pod \"ingress-canary-vgzzv\" (UID: \"3f60b419-0fe0-4291-a2be-eea12b95a646\") " pod="openshift-ingress-canary/ingress-canary-vgzzv"
Jul 06 16:29:18.232368 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.232330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vgzzv"
Jul 06 16:29:18.611714 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:18.611676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:18.611878 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:18.611817    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:29:18.611926 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:18.611880    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:19.61186041 +0000 UTC m=+37.049371296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:29:19.182701 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:19.182487    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vgzzv"]
Jul 06 16:29:19.187856 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:29:19.187825    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f60b419_0fe0_4291_a2be_eea12b95a646.slice/crio-68dfe314d2adda98d3dc242ebda0d6db37b995fb7860d172507397daab6f5d94 WatchSource:0}: Error finding container 68dfe314d2adda98d3dc242ebda0d6db37b995fb7860d172507397daab6f5d94: Status 404 returned error can't find the container with id 68dfe314d2adda98d3dc242ebda0d6db37b995fb7860d172507397daab6f5d94
Jul 06 16:29:19.281924 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:19.281894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vgzzv" event={"ID":"3f60b419-0fe0-4291-a2be-eea12b95a646","Type":"ContainerStarted","Data":"68dfe314d2adda98d3dc242ebda0d6db37b995fb7860d172507397daab6f5d94"}
Jul 06 16:29:19.287813 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:19.287783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerStarted","Data":"d5ca6576b7b7cf77594c4171b1c26f7c73081d3eaf6921a45b7a54ce20974b01"}
Jul 06 16:29:19.620081 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:19.620044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:19.620221 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:19.620155    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:29:19.620221 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:19.620205    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:21.620192449 +0000 UTC m=+39.057703336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:29:20.293022 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:20.292987    2578 generic.go:356] "Generic (PLEG): container finished" podID="05388e7b-2246-4681-9c88-00caef6231de" containerID="d5ca6576b7b7cf77594c4171b1c26f7c73081d3eaf6921a45b7a54ce20974b01" exitCode=0
Jul 06 16:29:20.293414 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:20.293036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerDied","Data":"d5ca6576b7b7cf77594c4171b1c26f7c73081d3eaf6921a45b7a54ce20974b01"}
Jul 06 16:29:21.299080 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:21.299043    2578 generic.go:356] "Generic (PLEG): container finished" podID="05388e7b-2246-4681-9c88-00caef6231de" containerID="3a3b53980a97804dd64a6d26d3941cc9769c1a893bb158d09ef0913c74dd6d28" exitCode=0
Jul 06 16:29:21.299451 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:21.299104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerDied","Data":"3a3b53980a97804dd64a6d26d3941cc9769c1a893bb158d09ef0913c74dd6d28"}
Jul 06 16:29:21.634926 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:21.634789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:21.635104 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:21.634975    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:29:21.635104 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:21.635057    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:25.635035441 +0000 UTC m=+43.072546331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:29:22.304887 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:22.304850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" event={"ID":"05388e7b-2246-4681-9c88-00caef6231de","Type":"ContainerStarted","Data":"98c02b1f7ed04ee873a8844b589e3b95bd18b8cf8cc13f92037371e4437d7f6e"}
Jul 06 16:29:22.306338 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:22.306310    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vgzzv" event={"ID":"3f60b419-0fe0-4291-a2be-eea12b95a646","Type":"ContainerStarted","Data":"41e288c0eb3b18ea5925304ca870200c2b03af5185e909d2b8c9efbfae7ff8e3"}
Jul 06 16:29:22.331167 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:22.331125    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7p6kj" podStartSLOduration=13.028646924 podStartE2EDuration="39.331109041s" totalImagesPullingTime="26.302462117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:22.330724694 +0000 UTC m=+39.768235603" watchObservedRunningTime="2026-07-06 16:29:22.331109041 +0000 UTC m=+39.768619950"
Jul 06 16:29:22.347464 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:22.347412    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vgzzv" podStartSLOduration=2.568137024 podStartE2EDuration="5.347393656s" totalImagesPullingTime="2.779256632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:22.346866783 +0000 UTC m=+39.784377691" watchObservedRunningTime="2026-07-06 16:29:22.347393656 +0000 UTC m=+39.784904565"
Jul 06 16:29:25.662050 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:25.661857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:25.662474 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:25.662016    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:29:25.662474 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:25.662167    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:33.662151359 +0000 UTC m=+51.099662245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:29:33.724116 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:33.724079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:33.724698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:33.724192    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:29:33.724698 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:33.724244    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:29:49.724229806 +0000 UTC m=+67.161740692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:29:45.281554 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:45.281523    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j748n"
Jul 06 16:29:48.830759 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.830722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:48.831231 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.830769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:48.831231 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.830894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:29:48.833646 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.833609    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 16:29:48.833646 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.833623    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 16:29:48.833820 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.833623    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 16:29:48.841905 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:48.841877    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 16:29:48.842055 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:48.842011    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:30:52.841992441 +0000 UTC m=+130.279503327 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : secret "metrics-daemon-secret" not found
Jul 06 16:29:48.844432 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.844414    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 16:29:48.845856 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.845835    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36dc0470-269b-4bed-9ab1-f1c0e63b2440-original-pull-secret\") pod \"global-pull-secret-syncer-slpn6\" (UID: \"36dc0470-269b-4bed-9ab1-f1c0e63b2440\") " pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:48.855152 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:48.855124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9x9g\" (UniqueName: \"kubernetes.io/projected/1bef6b96-c17a-4f17-8767-62750be0051a-kube-api-access-v9x9g\") pod \"network-check-target-skj5h\" (UID: \"1bef6b96-c17a-4f17-8767-62750be0051a\") " pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:49.031478 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:49.031441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slpn6"
Jul 06 16:29:49.047735 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:49.047709    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jqc2r\""
Jul 06 16:29:49.055084 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:49.055052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:49.168327 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:49.168296    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-slpn6"]
Jul 06 16:29:49.170906 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:29:49.170877    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dc0470_269b_4bed_9ab1_f1c0e63b2440.slice/crio-3edf7d83125401e28f20d3861fa7b70f43419e0b5953237e49a93aced57faaca WatchSource:0}: Error finding container 3edf7d83125401e28f20d3861fa7b70f43419e0b5953237e49a93aced57faaca: Status 404 returned error can't find the container with id 3edf7d83125401e28f20d3861fa7b70f43419e0b5953237e49a93aced57faaca
Jul 06 16:29:49.185852 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:49.185824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-skj5h"]
Jul 06 16:29:49.188661 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:29:49.188636    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bef6b96_c17a_4f17_8767_62750be0051a.slice/crio-8095ebc27a25d2ced47e101ecf39064c994bfaa75c77cf90936beaebf0366066 WatchSource:0}: Error finding container 8095ebc27a25d2ced47e101ecf39064c994bfaa75c77cf90936beaebf0366066: Status 404 returned error can't find the container with id 8095ebc27a25d2ced47e101ecf39064c994bfaa75c77cf90936beaebf0366066
Jul 06 16:29:49.369214 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:49.369123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-skj5h" event={"ID":"1bef6b96-c17a-4f17-8767-62750be0051a","Type":"ContainerStarted","Data":"8095ebc27a25d2ced47e101ecf39064c994bfaa75c77cf90936beaebf0366066"}
Jul 06 16:29:49.370156 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:49.370118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-slpn6" event={"ID":"36dc0470-269b-4bed-9ab1-f1c0e63b2440","Type":"ContainerStarted","Data":"3edf7d83125401e28f20d3861fa7b70f43419e0b5953237e49a93aced57faaca"}
Jul 06 16:29:49.737611 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:49.737527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:29:49.737786 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:49.737677    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:29:49.737786 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:29:49.737764    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:30:21.7377418 +0000 UTC m=+99.175252687 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:29:54.384471 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:54.384436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-skj5h" event={"ID":"1bef6b96-c17a-4f17-8767-62750be0051a","Type":"ContainerStarted","Data":"a651a548baa053c69fe91e00f37622775a4192d304117f9303ad0f2f5901b94d"}
Jul 06 16:29:54.384954 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:54.384555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:29:54.385919 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:54.385890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-slpn6" event={"ID":"36dc0470-269b-4bed-9ab1-f1c0e63b2440","Type":"ContainerStarted","Data":"094b294c06720f74c84c1111973948894492daa43b8d402843d31448f72712d2"}
Jul 06 16:29:54.400210 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:54.400166    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-skj5h" podStartSLOduration=66.719300453 podStartE2EDuration="1m11.400150644s" totalImagesPullingTime="4.680850191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:54.400007218 +0000 UTC m=+71.837518128" watchObservedRunningTime="2026-07-06 16:29:54.400150644 +0000 UTC m=+71.837661581"
Jul 06 16:29:55.756642 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.756588    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-slpn6" podStartSLOduration=68.053522161 podStartE2EDuration="1m12.756569434s" totalImagesPullingTime="4.703047273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:54.414229334 +0000 UTC m=+71.851740266" watchObservedRunningTime="2026-07-06 16:29:55.756569434 +0000 UTC m=+73.194080342"
Jul 06 16:29:55.757055 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.756868    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"]
Jul 06 16:29:55.758995 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.758978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.762811 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.762787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 16:29:55.763425 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.763408    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 16:29:55.764132 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.764051    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 16:29:55.764339 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.764321    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 16:29:55.766927 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.766908    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 16:29:55.768148 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.768134    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 16:29:55.772258 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.772243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 16:29:55.778461 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.778436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-ca\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.778574 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.778476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.778574 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.778529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.778574 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.778549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-hub\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.778668 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.778573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.778668 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.778626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdfp\" (UniqueName: \"kubernetes.io/projected/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-kube-api-access-vfdfp\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.786585 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.786563    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"]
Jul 06 16:29:55.879731 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.879690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-ca\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.879731 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.879725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.879974 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.879745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.879974 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.879762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-hub\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.879974 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.879783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.879974 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.879828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdfp\" (UniqueName: \"kubernetes.io/projected/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-kube-api-access-vfdfp\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.880820 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.880791    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.882357 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.882334    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-ca\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.882462 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.882392    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-hub\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.882596 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.882572    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.882652 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.882639    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:55.889145 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:55.889126    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdfp\" (UniqueName: \"kubernetes.io/projected/1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4-kube-api-access-vfdfp\") pod \"cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb\" (UID: \"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:56.079413 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:56.079383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:29:56.207336 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:56.207303    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"]
Jul 06 16:29:56.210678 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:29:56.210648    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1667c4b6_8a3c_43fb_ad83_f8b536cd0fc4.slice/crio-920478b01c1f51bf9fa1df5fd3b63edf1e59b36a1e89f09530813f7937684bc0 WatchSource:0}: Error finding container 920478b01c1f51bf9fa1df5fd3b63edf1e59b36a1e89f09530813f7937684bc0: Status 404 returned error can't find the container with id 920478b01c1f51bf9fa1df5fd3b63edf1e59b36a1e89f09530813f7937684bc0
Jul 06 16:29:56.391898 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:29:56.391809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" event={"ID":"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4","Type":"ContainerStarted","Data":"920478b01c1f51bf9fa1df5fd3b63edf1e59b36a1e89f09530813f7937684bc0"}
Jul 06 16:30:00.406068 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:30:00.406029    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" event={"ID":"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4","Type":"ContainerStarted","Data":"8104f09d0ac128d93ced1f35f63d6835a57e256ef0ba9a7713860bc3e560ca68"}
Jul 06 16:30:00.406068 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:30:00.406073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" event={"ID":"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4","Type":"ContainerStarted","Data":"79e5bd70e9e7db6205595d37ee31eceffdfa4d4df09615f3c2eecf65513ff446"}
Jul 06 16:30:00.406583 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:30:00.406087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" event={"ID":"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4","Type":"ContainerStarted","Data":"3dd2647a9f8bc39374e919e3eef00b80493ad18c6df57ab7a2d34448c3e67be3"}
Jul 06 16:30:00.425951 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:30:00.425891    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" podStartSLOduration=2.161162045 podStartE2EDuration="5.425870658s" totalImagesPullingTime="3.264708613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:30:00.425456763 +0000 UTC m=+77.862967673" watchObservedRunningTime="2026-07-06 16:30:00.425870658 +0000 UTC m=+77.863381571"
Jul 06 16:30:21.769707 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:30:21.769563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:30:21.770281 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:30:21.769740    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:30:21.770281 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:30:21.769839    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:31:25.769814229 +0000 UTC m=+163.207325124 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:30:25.390081 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:30:25.390049    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-skj5h"
Jul 06 16:30:52.885009 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:30:52.884961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:30:52.885644 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:30:52.885113    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 16:30:52.885644 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:30:52.885190    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs podName:681e3ec4-fa09-4f46-b9f1-543d1d3a816e nodeName:}" failed. No retries permitted until 2026-07-06 16:32:54.88517355 +0000 UTC m=+252.322684436 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs") pod "network-metrics-daemon-692bf" (UID: "681e3ec4-fa09-4f46-b9f1-543d1d3a816e") : secret "metrics-daemon-secret" not found
Jul 06 16:31:20.925600 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:31:20.925542    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2nbjn" podUID="685eb0d5-e105-422a-96fd-ff597ea88343"
Jul 06 16:31:21.138875 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:31:21.138831    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-692bf" podUID="681e3ec4-fa09-4f46-b9f1-543d1d3a816e"
Jul 06 16:31:21.635402 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:31:21.635375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:31:25.817076 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:31:25.817037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:31:25.817453 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:31:25.817143    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 16:31:25.817453 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:31:25.817202    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls podName:685eb0d5-e105-422a-96fd-ff597ea88343 nodeName:}" failed. No retries permitted until 2026-07-06 16:33:27.817189098 +0000 UTC m=+285.254699985 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls") pod "dns-default-2nbjn" (UID: "685eb0d5-e105-422a-96fd-ff597ea88343") : secret "dns-default-metrics-tls" not found
Jul 06 16:31:35.106574 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:31:35.106534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:32:29.777057 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.777023    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74"]
Jul 06 16:32:29.779057 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.779042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74"
Jul 06 16:32:29.781649 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.781623    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 06 16:32:29.781757 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.781627    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 06 16:32:29.781757 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.781709    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hrx72\""
Jul 06 16:32:29.788965 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.788943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74"]
Jul 06 16:32:29.840695 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.840666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpvz\" (UniqueName: \"kubernetes.io/projected/0e8b968e-950c-453a-a4e9-2c5281824706-kube-api-access-jhpvz\") pod \"migrator-d57d58cc9-zsj74\" (UID: \"0e8b968e-950c-453a-a4e9-2c5281824706\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74"
Jul 06 16:32:29.941231 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.941200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpvz\" (UniqueName: \"kubernetes.io/projected/0e8b968e-950c-453a-a4e9-2c5281824706-kube-api-access-jhpvz\") pod \"migrator-d57d58cc9-zsj74\" (UID: \"0e8b968e-950c-453a-a4e9-2c5281824706\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74"
Jul 06 16:32:29.950256 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:29.950225    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpvz\" (UniqueName: \"kubernetes.io/projected/0e8b968e-950c-453a-a4e9-2c5281824706-kube-api-access-jhpvz\") pod \"migrator-d57d58cc9-zsj74\" (UID: \"0e8b968e-950c-453a-a4e9-2c5281824706\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74"
Jul 06 16:32:30.087375 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.087341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74"
Jul 06 16:32:30.207817 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.207789    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74"]
Jul 06 16:32:30.210627 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:32:30.210601    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8b968e_950c_453a_a4e9_2c5281824706.slice/crio-96db49f8bce7d393ca157942543e796c59ccd77214743e33f042f941ac531e06 WatchSource:0}: Error finding container 96db49f8bce7d393ca157942543e796c59ccd77214743e33f042f941ac531e06: Status 404 returned error can't find the container with id 96db49f8bce7d393ca157942543e796c59ccd77214743e33f042f941ac531e06
Jul 06 16:32:30.823482 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.823442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74" event={"ID":"0e8b968e-950c-453a-a4e9-2c5281824706","Type":"ContainerStarted","Data":"96db49f8bce7d393ca157942543e796c59ccd77214743e33f042f941ac531e06"}
Jul 06 16:32:30.876926 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.876899    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d9wvk"]
Jul 06 16:32:30.879933 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.879913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:30.882464 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.882438    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 16:32:30.882596 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.882572    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 16:32:30.882665 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.882572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 16:32:30.882994 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.882980    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 16:32:30.883758 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.883745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-285th\""
Jul 06 16:32:30.894084 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.894065    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d9wvk"]
Jul 06 16:32:30.949428 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.949396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:30.949633 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.949456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntsh\" (UniqueName: \"kubernetes.io/projected/51611d17-8693-4b9b-ba1d-084079679f98-kube-api-access-hntsh\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:30.949633 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.949536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:30.949633 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.949595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51611d17-8693-4b9b-ba1d-084079679f98-data-volume\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:30.949756 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:30.949650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51611d17-8693-4b9b-ba1d-084079679f98-crio-socket\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.050357 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.050319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntsh\" (UniqueName: \"kubernetes.io/projected/51611d17-8693-4b9b-ba1d-084079679f98-kube-api-access-hntsh\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.050357 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.050361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.050565 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.050401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51611d17-8693-4b9b-ba1d-084079679f98-data-volume\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.050565 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.050423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51611d17-8693-4b9b-ba1d-084079679f98-crio-socket\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.050565 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.050441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.050565 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:31.050535    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 16:32:31.050742 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.050573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51611d17-8693-4b9b-ba1d-084079679f98-crio-socket\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.050742 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:31.050594    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:31.550579331 +0000 UTC m=+228.988090222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : configmap "kube-rbac-proxy" not found
Jul 06 16:32:31.050742 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:31.050605    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:31.050742 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:31.050683    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:31.550662457 +0000 UTC m=+228.988173343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:31.050868 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.050769    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51611d17-8693-4b9b-ba1d-084079679f98-data-volume\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.060180 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.060158    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntsh\" (UniqueName: \"kubernetes.io/projected/51611d17-8693-4b9b-ba1d-084079679f98-kube-api-access-hntsh\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.553927 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.553873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.553927 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:31.553941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:31.554200 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:31.554021    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 16:32:31.554200 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:31.554098    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:32.554077935 +0000 UTC m=+229.991588823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : configmap "kube-rbac-proxy" not found
Jul 06 16:32:31.554200 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:31.554028    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:31.554200 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:31.554154    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:32.554143125 +0000 UTC m=+229.991654015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:32.560268 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:32.560233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:32.560685 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:32.560292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:32.560685 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:32.560354    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 16:32:32.560685 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:32.560385    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:32.560685 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:32.560401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:34.560388192 +0000 UTC m=+231.997899079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : configmap "kube-rbac-proxy" not found
Jul 06 16:32:32.560685 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:32.560436    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:34.560422553 +0000 UTC m=+231.997933440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:32.831801 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:32.831769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74" event={"ID":"0e8b968e-950c-453a-a4e9-2c5281824706","Type":"ContainerStarted","Data":"a2ac0b228a0b6eca0bfd77a6592e964fadcfd951172cd28c4816a6c7db3f53ce"}
Jul 06 16:32:32.831961 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:32.831819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74" event={"ID":"0e8b968e-950c-453a-a4e9-2c5281824706","Type":"ContainerStarted","Data":"5cbdef4a33528081f0414dfc7240d6d805fab7a55b2e02aa1c39f04f1816b248"}
Jul 06 16:32:32.849789 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:32.849740    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zsj74" podStartSLOduration=1.7067782679999999 podStartE2EDuration="3.849723934s" totalImagesPullingTime="2.142945666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:32:32.849408988 +0000 UTC m=+230.286919894" watchObservedRunningTime="2026-07-06 16:32:32.849723934 +0000 UTC m=+230.287234843"
Jul 06 16:32:33.475604 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:33.475548    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8k9x7_22e56b49-6747-48b3-891d-09f7547abfe4/dns-node-resolver/0.log"
Jul 06 16:32:34.573615 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:34.573558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:34.574045 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:34.573643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:34.574045 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:34.573710    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 16:32:34.574045 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:34.573719    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:34.574045 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:34.573762    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:38.573748139 +0000 UTC m=+236.011259026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : configmap "kube-rbac-proxy" not found
Jul 06 16:32:34.574045 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:34.573775    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:38.573769819 +0000 UTC m=+236.011280707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:34.875877 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:34.875801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-smb7q_f8de9897-da9a-4c99-bd4e-9bdfab9d037f/node-ca/0.log"
Jul 06 16:32:35.676007 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:35.675978    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vgzzv_3f60b419-0fe0-4291-a2be-eea12b95a646/serve-healthcheck-canary/0.log"
Jul 06 16:32:35.876721 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:35.876694    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-zsj74_0e8b968e-950c-453a-a4e9-2c5281824706/migrator/0.log"
Jul 06 16:32:36.077409 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:36.077378    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-zsj74_0e8b968e-950c-453a-a4e9-2c5281824706/graceful-termination/0.log"
Jul 06 16:32:38.599437 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:38.599407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:38.599884 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:38.599455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:38.599884 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:38.599563    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 16:32:38.599884 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:38.599572    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:38.599884 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:38.599626    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:46.599608496 +0000 UTC m=+244.037119382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : configmap "kube-rbac-proxy" not found
Jul 06 16:32:38.599884 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:38.599640    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:46.59963319 +0000 UTC m=+244.037144076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:46.657077 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:46.657039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:46.657493 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:46.657099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:46.657493 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:46.657196    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:46.657493 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:32:46.657265    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls podName:51611d17-8693-4b9b-ba1d-084079679f98 nodeName:}" failed. No retries permitted until 2026-07-06 16:33:02.657245487 +0000 UTC m=+260.094756374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9wvk" (UID: "51611d17-8693-4b9b-ba1d-084079679f98") : secret "insights-runtime-extractor-tls" not found
Jul 06 16:32:46.657627 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:46.657574    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51611d17-8693-4b9b-ba1d-084079679f98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:32:52.619015 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.618983    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76dd8f5875-hqdg8"]
Jul 06 16:32:52.621628 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.621611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.624668 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.624639    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 16:32:52.624799 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.624742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 16:32:52.626151 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.626114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 16:32:52.626151 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.626142    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hmcqv\""
Jul 06 16:32:52.630418 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.630399    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 16:32:52.640206 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.640180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76dd8f5875-hqdg8"]
Jul 06 16:32:52.701989 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.701949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84d6e93c-2bf6-46b9-9632-8396ec425388-ca-trust-extracted\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.701989 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.701990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84d6e93c-2bf6-46b9-9632-8396ec425388-installation-pull-secrets\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.702263 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.702108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-registry-tls\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.702263 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.702139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84d6e93c-2bf6-46b9-9632-8396ec425388-registry-certificates\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.702263 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.702157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84d6e93c-2bf6-46b9-9632-8396ec425388-trusted-ca\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.702263 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.702200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84d6e93c-2bf6-46b9-9632-8396ec425388-image-registry-private-configuration\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.702263 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.702237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j84n\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-kube-api-access-9j84n\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.702412 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.702265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-bound-sa-token\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.803378 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.803334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84d6e93c-2bf6-46b9-9632-8396ec425388-image-registry-private-configuration\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.803378 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.803369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j84n\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-kube-api-access-9j84n\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.803378 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.803387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-bound-sa-token\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.803722 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.803599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84d6e93c-2bf6-46b9-9632-8396ec425388-ca-trust-extracted\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.803722 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.803621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84d6e93c-2bf6-46b9-9632-8396ec425388-installation-pull-secrets\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.803722 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.803668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-registry-tls\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.803864 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.803822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84d6e93c-2bf6-46b9-9632-8396ec425388-registry-certificates\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.803920 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.803871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84d6e93c-2bf6-46b9-9632-8396ec425388-trusted-ca\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.804049 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.804017    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84d6e93c-2bf6-46b9-9632-8396ec425388-ca-trust-extracted\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.804895 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.804861    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84d6e93c-2bf6-46b9-9632-8396ec425388-registry-certificates\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.805230 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.805208    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84d6e93c-2bf6-46b9-9632-8396ec425388-trusted-ca\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.806024 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.805994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84d6e93c-2bf6-46b9-9632-8396ec425388-image-registry-private-configuration\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.806136 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.806108    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84d6e93c-2bf6-46b9-9632-8396ec425388-installation-pull-secrets\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.806472 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.806452    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-registry-tls\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.815646 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.815609    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-bound-sa-token\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.815770 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.815750    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j84n\" (UniqueName: \"kubernetes.io/projected/84d6e93c-2bf6-46b9-9632-8396ec425388-kube-api-access-9j84n\") pod \"image-registry-76dd8f5875-hqdg8\" (UID: \"84d6e93c-2bf6-46b9-9632-8396ec425388\") " pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:52.931549 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:52.931437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:53.061710 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:53.061679    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76dd8f5875-hqdg8"]
Jul 06 16:32:53.063974 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:32:53.063940    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d6e93c_2bf6_46b9_9632_8396ec425388.slice/crio-9a829487cfdda306784b2cedfa1c379259c6fef5ec28260f5a4d679cd8784e75 WatchSource:0}: Error finding container 9a829487cfdda306784b2cedfa1c379259c6fef5ec28260f5a4d679cd8784e75: Status 404 returned error can't find the container with id 9a829487cfdda306784b2cedfa1c379259c6fef5ec28260f5a4d679cd8784e75
Jul 06 16:32:53.888022 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:53.887990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8" event={"ID":"84d6e93c-2bf6-46b9-9632-8396ec425388","Type":"ContainerStarted","Data":"558d39408abce334af1679e9c05f1d0a52dc457db728b63a7647069476ba56dd"}
Jul 06 16:32:53.888415 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:53.888036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8" event={"ID":"84d6e93c-2bf6-46b9-9632-8396ec425388","Type":"ContainerStarted","Data":"9a829487cfdda306784b2cedfa1c379259c6fef5ec28260f5a4d679cd8784e75"}
Jul 06 16:32:53.888415 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:53.888114    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:32:53.928679 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:53.928631    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8" podStartSLOduration=1.928613564 podStartE2EDuration="1.928613564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:32:53.928282919 +0000 UTC m=+251.365793828" watchObservedRunningTime="2026-07-06 16:32:53.928613564 +0000 UTC m=+251.366124472"
Jul 06 16:32:54.922201 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:54.922164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:32:54.924684 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:54.924651    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681e3ec4-fa09-4f46-b9f1-543d1d3a816e-metrics-certs\") pod \"network-metrics-daemon-692bf\" (UID: \"681e3ec4-fa09-4f46-b9f1-543d1d3a816e\") " pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:32:55.209927 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:55.209844    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jhwls\""
Jul 06 16:32:55.217238 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:55.217212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-692bf"
Jul 06 16:32:55.349325 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:55.349294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-692bf"]
Jul 06 16:32:55.351979 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:32:55.351949    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681e3ec4_fa09_4f46_b9f1_543d1d3a816e.slice/crio-d274aaed2567bc31747fb9f63acdd98cc32e6af4d6002eeab12271ae45630220 WatchSource:0}: Error finding container d274aaed2567bc31747fb9f63acdd98cc32e6af4d6002eeab12271ae45630220: Status 404 returned error can't find the container with id d274aaed2567bc31747fb9f63acdd98cc32e6af4d6002eeab12271ae45630220
Jul 06 16:32:55.896196 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:55.896165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-692bf" event={"ID":"681e3ec4-fa09-4f46-b9f1-543d1d3a816e","Type":"ContainerStarted","Data":"d274aaed2567bc31747fb9f63acdd98cc32e6af4d6002eeab12271ae45630220"}
Jul 06 16:32:56.900458 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:56.900414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-692bf" event={"ID":"681e3ec4-fa09-4f46-b9f1-543d1d3a816e","Type":"ContainerStarted","Data":"144d33338a231f0cdbcbeb07eeab8794d975ad5cc453ec531bf7a0970ade7471"}
Jul 06 16:32:56.900962 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:56.900482    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-692bf" event={"ID":"681e3ec4-fa09-4f46-b9f1-543d1d3a816e","Type":"ContainerStarted","Data":"329dedc183d34152ae3c8baa8be3627a0499ca14b60143afd53dc9e59c2e84d8"}
Jul 06 16:32:56.918521 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:32:56.918449    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-692bf" podStartSLOduration=252.722877568 podStartE2EDuration="4m13.918431922s" totalImagesPullingTime="1.195554354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:32:56.917966984 +0000 UTC m=+254.355477892" watchObservedRunningTime="2026-07-06 16:32:56.918431922 +0000 UTC m=+254.355942872"
Jul 06 16:33:02.684669 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:02.684622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:33:02.687066 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:02.687043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51611d17-8693-4b9b-ba1d-084079679f98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9wvk\" (UID: \"51611d17-8693-4b9b-ba1d-084079679f98\") " pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:33:02.691661 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:02.691638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-285th\""
Jul 06 16:33:02.699737 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:02.699718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d9wvk"
Jul 06 16:33:02.835101 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:02.835070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d9wvk"]
Jul 06 16:33:02.838391 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:33:02.838363    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51611d17_8693_4b9b_ba1d_084079679f98.slice/crio-f05d14e5b6506a4cb9c7a56bd902893a15da2ffc79acf7181e08dc8b5fb6cdcf WatchSource:0}: Error finding container f05d14e5b6506a4cb9c7a56bd902893a15da2ffc79acf7181e08dc8b5fb6cdcf: Status 404 returned error can't find the container with id f05d14e5b6506a4cb9c7a56bd902893a15da2ffc79acf7181e08dc8b5fb6cdcf
Jul 06 16:33:02.919104 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:02.919067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9wvk" event={"ID":"51611d17-8693-4b9b-ba1d-084079679f98","Type":"ContainerStarted","Data":"f05d14e5b6506a4cb9c7a56bd902893a15da2ffc79acf7181e08dc8b5fb6cdcf"}
Jul 06 16:33:07.938034 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:07.938008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9wvk" event={"ID":"51611d17-8693-4b9b-ba1d-084079679f98","Type":"ContainerStarted","Data":"4a9fb95a4856b75e5bb16e4da06bb77ead6fd173783bbb51896c93503157b488"}
Jul 06 16:33:07.938323 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:07.938083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9wvk" event={"ID":"51611d17-8693-4b9b-ba1d-084079679f98","Type":"ContainerStarted","Data":"edf06c1fee84e95d637a39781023a886ec9a27bfe2b3b5325a2b052b9f618dc4"}
Jul 06 16:33:10.945112 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.945077    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"]
Jul 06 16:33:10.948256 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.948234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:10.950489 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.950463    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7gt97"]
Jul 06 16:33:10.952714 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.952468    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 16:33:10.952714 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.952485    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 16:33:10.952714 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.952468    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 06 16:33:10.952714 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.952527    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 06 16:33:10.952714 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.952476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tggf5\""
Jul 06 16:33:10.952714 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.952476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 16:33:10.953026 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.952932    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 06 16:33:10.953981 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.953956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9wvk" event={"ID":"51611d17-8693-4b9b-ba1d-084079679f98","Type":"ContainerStarted","Data":"81d3b0e02d0b60afa61c5a3c869eb17bad2d3d359757eb03b07a02b7fc210981"}
Jul 06 16:33:10.954111 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.954097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:10.956647 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.956623    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 16:33:10.956785 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.956768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 16:33:10.956881 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.956868    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 16:33:10.957530 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.957487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-snrhp\""
Jul 06 16:33:10.961272 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:10.961252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"]
Jul 06 16:33:11.016233 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.016177    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d9wvk" podStartSLOduration=33.584134522 podStartE2EDuration="41.016161561s" totalImagesPullingTime="7.432027039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:32:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:33:11.015082073 +0000 UTC m=+268.452593002" watchObservedRunningTime="2026-07-06 16:33:11.016161561 +0000 UTC m=+268.453672471"
Jul 06 16:33:11.051518 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-wtmp\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.051669 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.051669 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-textfile\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.051669 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gkq\" (UniqueName: \"kubernetes.io/projected/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-api-access-w8gkq\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.051775 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-root\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.051775 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60fdec01-6df3-4101-aa1e-3063065d3c8a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.051775 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.051775 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.051933 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.051933 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.051933 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60fdec01-6df3-4101-aa1e-3063065d3c8a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.051933 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58b98e0f-01e5-4f18-9f4c-b28986786dcb-metrics-client-ca\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.052050 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlw4c\" (UniqueName: \"kubernetes.io/projected/58b98e0f-01e5-4f18-9f4c-b28986786dcb-kube-api-access-xlw4c\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.052050 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.052050 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.051984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-sys\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.152594 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60fdec01-6df3-4101-aa1e-3063065d3c8a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.152755 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.152755 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.152755 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.152755 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.152755 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60fdec01-6df3-4101-aa1e-3063065d3c8a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.152755 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:33:11.152745    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58b98e0f-01e5-4f18-9f4c-b28986786dcb-metrics-client-ca\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlw4c\" (UniqueName: \"kubernetes.io/projected/58b98e0f-01e5-4f18-9f4c-b28986786dcb-kube-api-access-xlw4c\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:33:11.152833    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls podName:58b98e0f-01e5-4f18-9f4c-b28986786dcb nodeName:}" failed. No retries permitted until 2026-07-06 16:33:11.652812077 +0000 UTC m=+269.090322976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls") pod "node-exporter-7gt97" (UID: "58b98e0f-01e5-4f18-9f4c-b28986786dcb") : secret "node-exporter-tls" not found
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-sys\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-wtmp\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.152979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-textfile\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153003    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60fdec01-6df3-4101-aa1e-3063065d3c8a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gkq\" (UniqueName: \"kubernetes.io/projected/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-api-access-w8gkq\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.153051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-root\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153120    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-root\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-wtmp\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153178    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58b98e0f-01e5-4f18-9f4c-b28986786dcb-sys\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153312    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153378    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-textfile\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.153793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153565    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58b98e0f-01e5-4f18-9f4c-b28986786dcb-metrics-client-ca\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.154101 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.153949    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60fdec01-6df3-4101-aa1e-3063065d3c8a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.154418 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.154386    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.155530 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.155486    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.156323 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.156296    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.156323 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.156316    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.162090 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.162068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlw4c\" (UniqueName: \"kubernetes.io/projected/58b98e0f-01e5-4f18-9f4c-b28986786dcb-kube-api-access-xlw4c\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.162191 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.162154    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gkq\" (UniqueName: \"kubernetes.io/projected/60fdec01-6df3-4101-aa1e-3063065d3c8a-kube-api-access-w8gkq\") pod \"kube-state-metrics-7c9dbb8b7d-6l9dj\" (UID: \"60fdec01-6df3-4101-aa1e-3063065d3c8a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.258859 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.258771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"
Jul 06 16:33:11.387695 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.387669    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj"]
Jul 06 16:33:11.390067 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:33:11.390040    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fdec01_6df3_4101_aa1e_3063065d3c8a.slice/crio-fa49dd099ec050796c5e7ed13332ee50c2bf7c770ff7958d75028ee56b526f35 WatchSource:0}: Error finding container fa49dd099ec050796c5e7ed13332ee50c2bf7c770ff7958d75028ee56b526f35: Status 404 returned error can't find the container with id fa49dd099ec050796c5e7ed13332ee50c2bf7c770ff7958d75028ee56b526f35
Jul 06 16:33:11.656851 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.656796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:11.657077 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:33:11.656984    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 06 16:33:11.657077 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:33:11.657063    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls podName:58b98e0f-01e5-4f18-9f4c-b28986786dcb nodeName:}" failed. No retries permitted until 2026-07-06 16:33:12.657043549 +0000 UTC m=+270.094554459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls") pod "node-exporter-7gt97" (UID: "58b98e0f-01e5-4f18-9f4c-b28986786dcb") : secret "node-exporter-tls" not found
Jul 06 16:33:11.955812 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:11.955713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj" event={"ID":"60fdec01-6df3-4101-aa1e-3063065d3c8a","Type":"ContainerStarted","Data":"fa49dd099ec050796c5e7ed13332ee50c2bf7c770ff7958d75028ee56b526f35"}
Jul 06 16:33:12.665903 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:12.665863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:12.668358 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:12.668336    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58b98e0f-01e5-4f18-9f4c-b28986786dcb-node-exporter-tls\") pod \"node-exporter-7gt97\" (UID: \"58b98e0f-01e5-4f18-9f4c-b28986786dcb\") " pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:12.765365 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:12.765327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7gt97"
Jul 06 16:33:12.893935 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:33:12.893905    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b98e0f_01e5_4f18_9f4c_b28986786dcb.slice/crio-f471cfcc3b00197eca550ef492c560a5ce4829ddb65748abee39fddbb26e02ff WatchSource:0}: Error finding container f471cfcc3b00197eca550ef492c560a5ce4829ddb65748abee39fddbb26e02ff: Status 404 returned error can't find the container with id f471cfcc3b00197eca550ef492c560a5ce4829ddb65748abee39fddbb26e02ff
Jul 06 16:33:12.935632 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:12.935605    2578 patch_prober.go:28] interesting pod/image-registry-76dd8f5875-hqdg8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 06 16:33:12.935738 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:12.935657    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8" podUID="84d6e93c-2bf6-46b9-9632-8396ec425388" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 16:33:12.959448 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:12.959416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gt97" event={"ID":"58b98e0f-01e5-4f18-9f4c-b28986786dcb","Type":"ContainerStarted","Data":"f471cfcc3b00197eca550ef492c560a5ce4829ddb65748abee39fddbb26e02ff"}
Jul 06 16:33:13.965040 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:13.965006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj" event={"ID":"60fdec01-6df3-4101-aa1e-3063065d3c8a","Type":"ContainerStarted","Data":"073ab90de4e43fd02a4a884da9c5828e61ee81bfd397d7fc2b57a3725372c9e6"}
Jul 06 16:33:13.965418 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:13.965060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj" event={"ID":"60fdec01-6df3-4101-aa1e-3063065d3c8a","Type":"ContainerStarted","Data":"e69377f7af8afc65589b2ef4ae143e0bbbfc389f8a8985dc3c62ab03ec36f211"}
Jul 06 16:33:13.965418 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:13.965080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj" event={"ID":"60fdec01-6df3-4101-aa1e-3063065d3c8a","Type":"ContainerStarted","Data":"07557ad1d4eedab400b612a3c89387501672c424eb0e35e2c31f103b1e13a477"}
Jul 06 16:33:13.986535 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:13.986466    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6l9dj" podStartSLOduration=2.445079531 podStartE2EDuration="3.986450169s" totalImagesPullingTime="1.541370638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:33:13.984961898 +0000 UTC m=+271.422472806" watchObservedRunningTime="2026-07-06 16:33:13.986450169 +0000 UTC m=+271.423961077"
Jul 06 16:33:14.897910 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:14.897882    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76dd8f5875-hqdg8"
Jul 06 16:33:15.224051 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.223973    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"]
Jul 06 16:33:15.226896 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.226878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.229577 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.229552    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 06 16:33:15.229577 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.229522    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 06 16:33:15.229827 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.229813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1g8b1lbnidtr6\""
Jul 06 16:33:15.229915 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.229897    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 06 16:33:15.229985 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.229927    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 06 16:33:15.229985 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.229954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tqh4s\""
Jul 06 16:33:15.239247 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.239224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"]
Jul 06 16:33:15.390200 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.390171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkln\" (UniqueName: \"kubernetes.io/projected/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-kube-api-access-jrkln\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.390372 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.390221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-secret-metrics-server-tls\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.390372 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.390248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.390372 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.390265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-metrics-server-audit-profiles\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.390372 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.390353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-secret-metrics-server-client-certs\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.390526 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.390377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-client-ca-bundle\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.390526 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.390397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-audit-log\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.491314 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.491236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-secret-metrics-server-client-certs\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.491314 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.491269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-client-ca-bundle\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.491314 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.491300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-audit-log\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.491607 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.491325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkln\" (UniqueName: \"kubernetes.io/projected/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-kube-api-access-jrkln\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.491607 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.491369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-secret-metrics-server-tls\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.491607 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.491405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.491607 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.491429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-metrics-server-audit-profiles\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.491817 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.491784    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-audit-log\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.492274 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.492254    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.492692 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.492671    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-metrics-server-audit-profiles\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.498528 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.494461    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-secret-metrics-server-client-certs\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.498528 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.494672    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-secret-metrics-server-tls\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.498528 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.494772    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-client-ca-bundle\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.500873 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.500849    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkln\" (UniqueName: \"kubernetes.io/projected/c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0-kube-api-access-jrkln\") pod \"metrics-server-58ccb5cf7f-h9lbh\" (UID: \"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0\") " pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.535577 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.535544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:15.667523 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.667370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"]
Jul 06 16:33:15.669991 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:33:15.669953    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82c16e7_f398_4fb4_8bc8_bf94fd2a41f0.slice/crio-44585df8aa41d9b2142861ed56090b140a7717c5f4fb835b896853d400887668 WatchSource:0}: Error finding container 44585df8aa41d9b2142861ed56090b140a7717c5f4fb835b896853d400887668: Status 404 returned error can't find the container with id 44585df8aa41d9b2142861ed56090b140a7717c5f4fb835b896853d400887668
Jul 06 16:33:15.695426 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.695396    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"]
Jul 06 16:33:15.699598 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.699578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:15.702587 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.702537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7ghn7\""
Jul 06 16:33:15.702752 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.702652    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 06 16:33:15.712464 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.712441    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"]
Jul 06 16:33:15.793735 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.793639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45c63935-cdc6-4034-8abe-8b1d0f4ac03e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-btg84\" (UID: \"45c63935-cdc6-4034-8abe-8b1d0f4ac03e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:15.793884 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.793748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8v87\" (UniqueName: \"kubernetes.io/projected/45c63935-cdc6-4034-8abe-8b1d0f4ac03e-kube-api-access-j8v87\") pod \"monitoring-plugin-d5dcff7cf-btg84\" (UID: \"45c63935-cdc6-4034-8abe-8b1d0f4ac03e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:15.894512 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.894478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45c63935-cdc6-4034-8abe-8b1d0f4ac03e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-btg84\" (UID: \"45c63935-cdc6-4034-8abe-8b1d0f4ac03e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:15.894667 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.894570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8v87\" (UniqueName: \"kubernetes.io/projected/45c63935-cdc6-4034-8abe-8b1d0f4ac03e-kube-api-access-j8v87\") pod \"monitoring-plugin-d5dcff7cf-btg84\" (UID: \"45c63935-cdc6-4034-8abe-8b1d0f4ac03e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:15.897037 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.897016    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45c63935-cdc6-4034-8abe-8b1d0f4ac03e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-btg84\" (UID: \"45c63935-cdc6-4034-8abe-8b1d0f4ac03e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:15.903188 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.903163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8v87\" (UniqueName: \"kubernetes.io/projected/45c63935-cdc6-4034-8abe-8b1d0f4ac03e-kube-api-access-j8v87\") pod \"monitoring-plugin-d5dcff7cf-btg84\" (UID: \"45c63935-cdc6-4034-8abe-8b1d0f4ac03e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:15.973967 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.973933    2578 generic.go:356] "Generic (PLEG): container finished" podID="58b98e0f-01e5-4f18-9f4c-b28986786dcb" containerID="b4ba0530f805a3277f9bb28e15416cadcb162b7d2e5bded614070cf4704b0251" exitCode=0
Jul 06 16:33:15.974160 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.974033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gt97" event={"ID":"58b98e0f-01e5-4f18-9f4c-b28986786dcb","Type":"ContainerDied","Data":"b4ba0530f805a3277f9bb28e15416cadcb162b7d2e5bded614070cf4704b0251"}
Jul 06 16:33:15.975250 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:15.975229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh" event={"ID":"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0","Type":"ContainerStarted","Data":"44585df8aa41d9b2142861ed56090b140a7717c5f4fb835b896853d400887668"}
Jul 06 16:33:16.008839 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:16.008813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:16.182581 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:16.182492    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"]
Jul 06 16:33:16.185210 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:33:16.185178    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c63935_cdc6_4034_8abe_8b1d0f4ac03e.slice/crio-dd482b6013e16b0daa65b3303cad8b417b015c5621953127ce813e9a84809e3f WatchSource:0}: Error finding container dd482b6013e16b0daa65b3303cad8b417b015c5621953127ce813e9a84809e3f: Status 404 returned error can't find the container with id dd482b6013e16b0daa65b3303cad8b417b015c5621953127ce813e9a84809e3f
Jul 06 16:33:16.982208 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:16.982162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gt97" event={"ID":"58b98e0f-01e5-4f18-9f4c-b28986786dcb","Type":"ContainerStarted","Data":"5050e4bca3496680d27a434ded2a5c8bdfb4cec89fc31a6af3d32ed972d926de"}
Jul 06 16:33:16.982764 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:16.982225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gt97" event={"ID":"58b98e0f-01e5-4f18-9f4c-b28986786dcb","Type":"ContainerStarted","Data":"5ac73ded5c26f32ef4da91c49f58990152d5353dfee27fcf337e775d6d0955cc"}
Jul 06 16:33:16.983641 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:16.983604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84" event={"ID":"45c63935-cdc6-4034-8abe-8b1d0f4ac03e","Type":"ContainerStarted","Data":"dd482b6013e16b0daa65b3303cad8b417b015c5621953127ce813e9a84809e3f"}
Jul 06 16:33:17.003704 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:17.003648    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7gt97" podStartSLOduration=4.850562269 podStartE2EDuration="7.003627172s" totalImagesPullingTime="2.079937366s" totalInitContainerRuntime="73.127537ms" isStatefulPod=true podCreationTimestamp="2026-07-06 16:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:33:17.001457083 +0000 UTC m=+274.438968017" watchObservedRunningTime="2026-07-06 16:33:17.003627172 +0000 UTC m=+274.441138083"
Jul 06 16:33:18.992122 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:18.992089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh" event={"ID":"c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0","Type":"ContainerStarted","Data":"c9148d723bbfbcf2e0cd9eb1430cf2a3fa26b2b9d69321cfba6fa5bc84fa12bf"}
Jul 06 16:33:18.993657 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:18.993637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84" event={"ID":"45c63935-cdc6-4034-8abe-8b1d0f4ac03e","Type":"ContainerStarted","Data":"4f8a6d27e32dd60bf35222208b9aa04668800784afbd18c57e707307abf09afd"}
Jul 06 16:33:18.993796 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:18.993781    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:18.998534 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:18.998515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84"
Jul 06 16:33:19.013172 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:19.013126    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh" podStartSLOduration=1.748765944 podStartE2EDuration="4.013106663s" totalImagesPullingTime="2.264340719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:33:19.01192118 +0000 UTC m=+276.449432089" watchObservedRunningTime="2026-07-06 16:33:19.013106663 +0000 UTC m=+276.450617573"
Jul 06 16:33:24.636130 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:33:24.636086    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2nbjn" podUID="685eb0d5-e105-422a-96fd-ff597ea88343"
Jul 06 16:33:25.014270 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:25.014194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:33:27.865884 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:27.865838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:33:27.868219 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:27.868199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685eb0d5-e105-422a-96fd-ff597ea88343-metrics-tls\") pod \"dns-default-2nbjn\" (UID: \"685eb0d5-e105-422a-96fd-ff597ea88343\") " pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:33:28.017457 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:28.017420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6rxcp\""
Jul 06 16:33:28.025233 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:28.025209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:33:28.169958 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:28.169861    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-btg84" podStartSLOduration=11.418026248 podStartE2EDuration="13.169842971s" totalImagesPullingTime="1.751816723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:33:19.031789877 +0000 UTC m=+276.469300787" watchObservedRunningTime="2026-07-06 16:33:28.169842971 +0000 UTC m=+285.607353880"
Jul 06 16:33:28.170553 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:28.170537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2nbjn"]
Jul 06 16:33:28.173054 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:33:28.173028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685eb0d5_e105_422a_96fd_ff597ea88343.slice/crio-db21982b46d9f6ffb9850f5c57c3af240837dd6c8253d2ae32a4ad29bb957df0 WatchSource:0}: Error finding container db21982b46d9f6ffb9850f5c57c3af240837dd6c8253d2ae32a4ad29bb957df0: Status 404 returned error can't find the container with id db21982b46d9f6ffb9850f5c57c3af240837dd6c8253d2ae32a4ad29bb957df0
Jul 06 16:33:29.029192 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:29.029154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nbjn" event={"ID":"685eb0d5-e105-422a-96fd-ff597ea88343","Type":"ContainerStarted","Data":"db21982b46d9f6ffb9850f5c57c3af240837dd6c8253d2ae32a4ad29bb957df0"}
Jul 06 16:33:30.034562 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:30.034523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nbjn" event={"ID":"685eb0d5-e105-422a-96fd-ff597ea88343","Type":"ContainerStarted","Data":"80474cb0ab6ec07a17cbc16e7e2cb26a6bd39f43cabc75228e903ebf6952c3ce"}
Jul 06 16:33:30.034942 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:30.034585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nbjn" event={"ID":"685eb0d5-e105-422a-96fd-ff597ea88343","Type":"ContainerStarted","Data":"58d9751859957bb50ee6a1c62dd2271ed23b7c3db89ca774ad982bc3427cf95e"}
Jul 06 16:33:30.034942 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:30.034629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:33:30.053146 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:30.053078    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2nbjn" podStartSLOduration=251.395261352 podStartE2EDuration="4m13.053055239s" totalImagesPullingTime="1.657793887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:33:30.052289384 +0000 UTC m=+287.489800320" watchObservedRunningTime="2026-07-06 16:33:30.053055239 +0000 UTC m=+287.490566150"
Jul 06 16:33:35.535775 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:35.535742    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:35.536246 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:35.535803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:40.040135 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:40.040105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2nbjn"
Jul 06 16:33:42.946942 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:42.946903    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 16:33:46.080805 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:46.080762    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" podUID="1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 16:33:55.540977 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:55.540942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:55.549159 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:55.549128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-58ccb5cf7f-h9lbh"
Jul 06 16:33:56.081171 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:33:56.081122    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" podUID="1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 16:34:06.080823 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:34:06.080783    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" podUID="1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 16:34:06.081321 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:34:06.080893    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb"
Jul 06 16:34:06.081542 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:34:06.081481    2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"8104f09d0ac128d93ced1f35f63d6835a57e256ef0ba9a7713860bc3e560ca68"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 06 16:34:06.081639 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:34:06.081569    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" podUID="1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4" containerName="service-proxy" containerID="cri-o://8104f09d0ac128d93ced1f35f63d6835a57e256ef0ba9a7713860bc3e560ca68" gracePeriod=30
Jul 06 16:34:07.164705 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:34:07.164672    2578 generic.go:356] "Generic (PLEG): container finished" podID="1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4" containerID="8104f09d0ac128d93ced1f35f63d6835a57e256ef0ba9a7713860bc3e560ca68" exitCode=2
Jul 06 16:34:07.165160 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:34:07.164711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" event={"ID":"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4","Type":"ContainerDied","Data":"8104f09d0ac128d93ced1f35f63d6835a57e256ef0ba9a7713860bc3e560ca68"}
Jul 06 16:34:07.165160 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:34:07.164738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fb9d4c6dd-s9ddb" event={"ID":"1667c4b6-8a3c-43fb-ad83-f8b536cd0fc4","Type":"ContainerStarted","Data":"d10640e33d0b7449a97d589c13497a7d17799473a032fdc3c2dd81381a0b947d"}
Jul 06 16:37:16.713728 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.713692    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"]
Jul 06 16:37:16.716776 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.716757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"
Jul 06 16:37:16.719527 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.719489    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 06 16:37:16.719649 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.719582    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 06 16:37:16.720693 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.720666    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-v4w7v\""
Jul 06 16:37:16.725784 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.725762    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"]
Jul 06 16:37:16.874268 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.874222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xnzh\" (UniqueName: \"kubernetes.io/projected/15a921f3-ff7c-4e33-8b50-7c8a409fdf9f-kube-api-access-6xnzh\") pod \"openshift-lws-operator-fd8ccff4c-qnxpb\" (UID: \"15a921f3-ff7c-4e33-8b50-7c8a409fdf9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"
Jul 06 16:37:16.874268 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.874270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15a921f3-ff7c-4e33-8b50-7c8a409fdf9f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qnxpb\" (UID: \"15a921f3-ff7c-4e33-8b50-7c8a409fdf9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"
Jul 06 16:37:16.974779 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.974683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnzh\" (UniqueName: \"kubernetes.io/projected/15a921f3-ff7c-4e33-8b50-7c8a409fdf9f-kube-api-access-6xnzh\") pod \"openshift-lws-operator-fd8ccff4c-qnxpb\" (UID: \"15a921f3-ff7c-4e33-8b50-7c8a409fdf9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"
Jul 06 16:37:16.974779 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.974733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15a921f3-ff7c-4e33-8b50-7c8a409fdf9f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qnxpb\" (UID: \"15a921f3-ff7c-4e33-8b50-7c8a409fdf9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"
Jul 06 16:37:16.975153 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.975133    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15a921f3-ff7c-4e33-8b50-7c8a409fdf9f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qnxpb\" (UID: \"15a921f3-ff7c-4e33-8b50-7c8a409fdf9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"
Jul 06 16:37:16.982647 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:16.982622    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnzh\" (UniqueName: \"kubernetes.io/projected/15a921f3-ff7c-4e33-8b50-7c8a409fdf9f-kube-api-access-6xnzh\") pod \"openshift-lws-operator-fd8ccff4c-qnxpb\" (UID: \"15a921f3-ff7c-4e33-8b50-7c8a409fdf9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"
Jul 06 16:37:17.026820 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:17.026778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"
Jul 06 16:37:17.162837 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:17.162815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb"]
Jul 06 16:37:17.165526 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:37:17.165484    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15a921f3_ff7c_4e33_8b50_7c8a409fdf9f.slice/crio-5c444ed6f4931fc9adcd2ad5c9fb77fc49d0da97d909cf73e00c725ca969aa9c WatchSource:0}: Error finding container 5c444ed6f4931fc9adcd2ad5c9fb77fc49d0da97d909cf73e00c725ca969aa9c: Status 404 returned error can't find the container with id 5c444ed6f4931fc9adcd2ad5c9fb77fc49d0da97d909cf73e00c725ca969aa9c
Jul 06 16:37:17.167489 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:17.167471    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 16:37:17.820394 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:17.820358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb" event={"ID":"15a921f3-ff7c-4e33-8b50-7c8a409fdf9f","Type":"ContainerStarted","Data":"5c444ed6f4931fc9adcd2ad5c9fb77fc49d0da97d909cf73e00c725ca969aa9c"}
Jul 06 16:37:21.837288 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:21.837251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb" event={"ID":"15a921f3-ff7c-4e33-8b50-7c8a409fdf9f","Type":"ContainerStarted","Data":"fa773b4137ebd60565cf79fb87b14d97f5618d7305c202d185f5cf77660e2aab"}
Jul 06 16:37:21.859597 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:21.859493    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qnxpb" podStartSLOduration=1.8712167 podStartE2EDuration="5.859474844s" totalImagesPullingTime="3.988258144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:37:21.853171645 +0000 UTC m=+519.290682722" watchObservedRunningTime="2026-07-06 16:37:21.859474844 +0000 UTC m=+519.296985753"
Jul 06 16:37:41.222779 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.222745    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"]
Jul 06 16:37:41.226586 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.226569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.231187 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.231163    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 06 16:37:41.231819 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.231787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 06 16:37:41.231968 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.231866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nfd8f\""
Jul 06 16:37:41.232058 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.232038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 06 16:37:41.232552 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.232531    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 06 16:37:41.251839 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.251804    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"]
Jul 06 16:37:41.260758 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.260729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de1e432a-07ea-40c7-84cb-f53bdc8afd55-manifests\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.260925 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.260764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de1e432a-07ea-40c7-84cb-f53bdc8afd55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.260925 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.260801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de1e432a-07ea-40c7-84cb-f53bdc8afd55-tmp\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.260925 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.260839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de1e432a-07ea-40c7-84cb-f53bdc8afd55-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.260925 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.260884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrww\" (UniqueName: \"kubernetes.io/projected/de1e432a-07ea-40c7-84cb-f53bdc8afd55-kube-api-access-5nrww\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.361844 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.361808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de1e432a-07ea-40c7-84cb-f53bdc8afd55-manifests\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.361844 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.361848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de1e432a-07ea-40c7-84cb-f53bdc8afd55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.362064 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.361874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de1e432a-07ea-40c7-84cb-f53bdc8afd55-tmp\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.362064 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.361895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de1e432a-07ea-40c7-84cb-f53bdc8afd55-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.362064 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.361929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrww\" (UniqueName: \"kubernetes.io/projected/de1e432a-07ea-40c7-84cb-f53bdc8afd55-kube-api-access-5nrww\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.362347 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.362313    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de1e432a-07ea-40c7-84cb-f53bdc8afd55-manifests\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.364515 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.364477    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de1e432a-07ea-40c7-84cb-f53bdc8afd55-tmp\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.364622 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.364588    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de1e432a-07ea-40c7-84cb-f53bdc8afd55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.364820 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.364798    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de1e432a-07ea-40c7-84cb-f53bdc8afd55-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.374296 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.374262    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrww\" (UniqueName: \"kubernetes.io/projected/de1e432a-07ea-40c7-84cb-f53bdc8afd55-kube-api-access-5nrww\") pod \"opendatahub-operator-controller-manager-bb6889b66-zj8vz\" (UID: \"de1e432a-07ea-40c7-84cb-f53bdc8afd55\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.536680 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.536585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:41.675481 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.675458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"]
Jul 06 16:37:41.677797 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:37:41.677775    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1e432a_07ea_40c7_84cb_f53bdc8afd55.slice/crio-6007f12944cd44dc258fc33be6c5f4c34391d0403cafef64ca3b3d7873d6d96d WatchSource:0}: Error finding container 6007f12944cd44dc258fc33be6c5f4c34391d0403cafef64ca3b3d7873d6d96d: Status 404 returned error can't find the container with id 6007f12944cd44dc258fc33be6c5f4c34391d0403cafef64ca3b3d7873d6d96d
Jul 06 16:37:41.916398 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:41.916360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz" event={"ID":"de1e432a-07ea-40c7-84cb-f53bdc8afd55","Type":"ContainerStarted","Data":"6007f12944cd44dc258fc33be6c5f4c34391d0403cafef64ca3b3d7873d6d96d"}
Jul 06 16:37:45.934096 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:45.934054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz" event={"ID":"de1e432a-07ea-40c7-84cb-f53bdc8afd55","Type":"ContainerStarted","Data":"e915c7390d7a144a005bce038830ad3afdd87640cf67faf2ec3f7b9ee9494fe7"}
Jul 06 16:37:46.938786 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:46.938748    2578 generic.go:356] "Generic (PLEG): container finished" podID="de1e432a-07ea-40c7-84cb-f53bdc8afd55" containerID="e915c7390d7a144a005bce038830ad3afdd87640cf67faf2ec3f7b9ee9494fe7" exitCode=0
Jul 06 16:37:46.939194 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:46.938836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz" event={"ID":"de1e432a-07ea-40c7-84cb-f53bdc8afd55","Type":"ContainerDied","Data":"e915c7390d7a144a005bce038830ad3afdd87640cf67faf2ec3f7b9ee9494fe7"}
Jul 06 16:37:49.951228 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:49.951193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz" event={"ID":"de1e432a-07ea-40c7-84cb-f53bdc8afd55","Type":"ContainerStarted","Data":"35a4efa85be447288f85d829c0b9e998e764217d0f0db33e35aa412f72e1bdba"}
Jul 06 16:37:49.951680 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:49.951436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:37:49.978624 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:37:49.978555    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz" podStartSLOduration=2.300491012 podStartE2EDuration="8.978534569s" totalImagesPullingTime="5.599702203s" totalInitContainerRuntime="1.078341354s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:37:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:37:49.9765815 +0000 UTC m=+547.414092409" watchObservedRunningTime="2026-07-06 16:37:49.978534569 +0000 UTC m=+547.416045479"
Jul 06 16:38:00.957272 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:00.957239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bb6889b66-zj8vz"
Jul 06 16:38:03.072972 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.072943    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"]
Jul 06 16:38:03.078476 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.078458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.082062 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.082039    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 16:38:03.083354 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.083334    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 06 16:38:03.083479 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.083459    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 16:38:03.083586 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.083563    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 06 16:38:03.083667 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.083563    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8sq6n\""
Jul 06 16:38:03.091966 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.091942    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"]
Jul 06 16:38:03.133632 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.133604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f5594-b381-4985-8881-611abb96b885-tls-certs\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.133779 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.133660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847f5594-b381-4985-8881-611abb96b885-tmp\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.133779 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.133720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9585s\" (UniqueName: \"kubernetes.io/projected/847f5594-b381-4985-8881-611abb96b885-kube-api-access-9585s\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.234708 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.234671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f5594-b381-4985-8881-611abb96b885-tls-certs\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.234708 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.234714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847f5594-b381-4985-8881-611abb96b885-tmp\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.234954 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.234756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9585s\" (UniqueName: \"kubernetes.io/projected/847f5594-b381-4985-8881-611abb96b885-kube-api-access-9585s\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.237221 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.237188    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847f5594-b381-4985-8881-611abb96b885-tmp\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.237346 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.237311    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f5594-b381-4985-8881-611abb96b885-tls-certs\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.249201 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.249165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9585s\" (UniqueName: \"kubernetes.io/projected/847f5594-b381-4985-8881-611abb96b885-kube-api-access-9585s\") pod \"kube-auth-proxy-75577b88c-wzwhh\" (UID: \"847f5594-b381-4985-8881-611abb96b885\") " pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.312971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.312933    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4svsh"]
Jul 06 16:38:03.315334 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.315315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:03.320724 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.320700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pcf6l\""
Jul 06 16:38:03.321204 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.321189    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 06 16:38:03.333671 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.333603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4svsh"]
Jul 06 16:38:03.387910 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.387879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"
Jul 06 16:38:03.436206 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.436055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfd9\" (UniqueName: \"kubernetes.io/projected/78049e17-82e0-46e7-a85d-27aaf3ffcf30-kube-api-access-7bfd9\") pod \"odh-model-controller-7998c67dc4-4svsh\" (UID: \"78049e17-82e0-46e7-a85d-27aaf3ffcf30\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:03.436206 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.436122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78049e17-82e0-46e7-a85d-27aaf3ffcf30-cert\") pod \"odh-model-controller-7998c67dc4-4svsh\" (UID: \"78049e17-82e0-46e7-a85d-27aaf3ffcf30\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:03.531672 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.531649    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75577b88c-wzwhh"]
Jul 06 16:38:03.533697 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:38:03.533669    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847f5594_b381_4985_8881_611abb96b885.slice/crio-56da484b6b6a54b91d8f7588d2273d3a7c26fbea3802611d6981e1edfb9f6694 WatchSource:0}: Error finding container 56da484b6b6a54b91d8f7588d2273d3a7c26fbea3802611d6981e1edfb9f6694: Status 404 returned error can't find the container with id 56da484b6b6a54b91d8f7588d2273d3a7c26fbea3802611d6981e1edfb9f6694
Jul 06 16:38:03.537218 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.537195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78049e17-82e0-46e7-a85d-27aaf3ffcf30-cert\") pod \"odh-model-controller-7998c67dc4-4svsh\" (UID: \"78049e17-82e0-46e7-a85d-27aaf3ffcf30\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:03.537319 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.537278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfd9\" (UniqueName: \"kubernetes.io/projected/78049e17-82e0-46e7-a85d-27aaf3ffcf30-kube-api-access-7bfd9\") pod \"odh-model-controller-7998c67dc4-4svsh\" (UID: \"78049e17-82e0-46e7-a85d-27aaf3ffcf30\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:03.537381 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:38:03.537369    2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 06 16:38:03.537449 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:38:03.537436    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78049e17-82e0-46e7-a85d-27aaf3ffcf30-cert podName:78049e17-82e0-46e7-a85d-27aaf3ffcf30 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:04.03741484 +0000 UTC m=+561.474925742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/78049e17-82e0-46e7-a85d-27aaf3ffcf30-cert") pod "odh-model-controller-7998c67dc4-4svsh" (UID: "78049e17-82e0-46e7-a85d-27aaf3ffcf30") : secret "odh-model-controller-webhook-cert" not found
Jul 06 16:38:03.547402 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:03.547377    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfd9\" (UniqueName: \"kubernetes.io/projected/78049e17-82e0-46e7-a85d-27aaf3ffcf30-kube-api-access-7bfd9\") pod \"odh-model-controller-7998c67dc4-4svsh\" (UID: \"78049e17-82e0-46e7-a85d-27aaf3ffcf30\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:04.001390 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.001354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh" event={"ID":"847f5594-b381-4985-8881-611abb96b885","Type":"ContainerStarted","Data":"56da484b6b6a54b91d8f7588d2273d3a7c26fbea3802611d6981e1edfb9f6694"}
Jul 06 16:38:04.041716 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.041685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78049e17-82e0-46e7-a85d-27aaf3ffcf30-cert\") pod \"odh-model-controller-7998c67dc4-4svsh\" (UID: \"78049e17-82e0-46e7-a85d-27aaf3ffcf30\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:04.044211 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.044186    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78049e17-82e0-46e7-a85d-27aaf3ffcf30-cert\") pod \"odh-model-controller-7998c67dc4-4svsh\" (UID: \"78049e17-82e0-46e7-a85d-27aaf3ffcf30\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:04.225641 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.225609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:04.383318 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.383257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4svsh"]
Jul 06 16:38:04.385406 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:38:04.385378    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78049e17_82e0_46e7_a85d_27aaf3ffcf30.slice/crio-d5de67efc83d1cb63da8fc07e6ce295fc1dc8baffe65020ff3d1ff03e639b483 WatchSource:0}: Error finding container d5de67efc83d1cb63da8fc07e6ce295fc1dc8baffe65020ff3d1ff03e639b483: Status 404 returned error can't find the container with id d5de67efc83d1cb63da8fc07e6ce295fc1dc8baffe65020ff3d1ff03e639b483
Jul 06 16:38:04.699874 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.699837    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"]
Jul 06 16:38:04.702342 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.702323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.705408 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.705389    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 16:38:04.705565 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.705430    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 06 16:38:04.705565 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.705396    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w7nwh\""
Jul 06 16:38:04.705565 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.705487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 06 16:38:04.705700 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.705689    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 06 16:38:04.717870 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.717844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"]
Jul 06 16:38:04.746971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.746937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.746971 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.746969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-cacerts\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.747204 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.746990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.747204 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.747016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-local-certs\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.747204 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.747038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.747204 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.747136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvms\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-kube-api-access-mlvms\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.747204 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.747182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-token\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.848439 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.848406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvms\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-kube-api-access-mlvms\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.848649 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.848452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-token\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.848649 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.848486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.848771 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.848684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-cacerts\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.848771 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.848732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.848870 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.848772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-local-certs\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.848870 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.848800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.849747 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.849694    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.851479 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.851455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-local-certs\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.851607 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.851517    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-cacerts\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.851650 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.851609    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.851650 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.851616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.857729 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.857594    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-token\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:04.857963 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:04.857945    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvms\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-kube-api-access-mlvms\") pod \"istiod-openshift-gateway-7574459656-kqszg\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:05.006649 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:05.006568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4svsh" event={"ID":"78049e17-82e0-46e7-a85d-27aaf3ffcf30","Type":"ContainerStarted","Data":"d5de67efc83d1cb63da8fc07e6ce295fc1dc8baffe65020ff3d1ff03e639b483"}
Jul 06 16:38:05.011751 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:05.011726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:05.147471 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:05.147443    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"]
Jul 06 16:38:05.149109 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:38:05.149079    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ed42a0_01d1_48bc_9cb0_51b150775b1c.slice/crio-d64300c5e0b46e8187e57da4123ba1e068a72fea35b99e05b2ea17e6f4465810 WatchSource:0}: Error finding container d64300c5e0b46e8187e57da4123ba1e068a72fea35b99e05b2ea17e6f4465810: Status 404 returned error can't find the container with id d64300c5e0b46e8187e57da4123ba1e068a72fea35b99e05b2ea17e6f4465810
Jul 06 16:38:06.012045 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:06.011992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg" event={"ID":"d3ed42a0-01d1-48bc-9cb0-51b150775b1c","Type":"ContainerStarted","Data":"d64300c5e0b46e8187e57da4123ba1e068a72fea35b99e05b2ea17e6f4465810"}
Jul 06 16:38:10.730807 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:10.730773    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:38:10.731180 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:10.730868    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:38:11.041266 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.041177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg" event={"ID":"d3ed42a0-01d1-48bc-9cb0-51b150775b1c","Type":"ContainerStarted","Data":"347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c"}
Jul 06 16:38:11.041449 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.041380    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:11.042872 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.042840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4svsh" event={"ID":"78049e17-82e0-46e7-a85d-27aaf3ffcf30","Type":"ContainerStarted","Data":"d978ade83cd81a4366e74be4a547a8f31cf731f2d1df094a8b910bc2b9535ed8"}
Jul 06 16:38:11.042987 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.042923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:11.043386 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.043363    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-kqszg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 16:38:11.043475 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.043405    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg" podUID="d3ed42a0-01d1-48bc-9cb0-51b150775b1c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 16:38:11.044394 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.044373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh" event={"ID":"847f5594-b381-4985-8881-611abb96b885","Type":"ContainerStarted","Data":"71200ad85b43c7d0d588b2fb492c223ab7c61fe3fba3e2ff0890110dc29a3a56"}
Jul 06 16:38:11.085572 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.085494    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg" podStartSLOduration=1.506843977 podStartE2EDuration="7.085472581s" totalImagesPullingTime="5.578628604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:11.083529307 +0000 UTC m=+568.521040208" watchObservedRunningTime="2026-07-06 16:38:11.085472581 +0000 UTC m=+568.522983492"
Jul 06 16:38:11.134943 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.134890    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4svsh" podStartSLOduration=1.785607706 podStartE2EDuration="8.134867896s" totalImagesPullingTime="6.34926019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:11.112125076 +0000 UTC m=+568.549636022" watchObservedRunningTime="2026-07-06 16:38:11.134867896 +0000 UTC m=+568.572378854"
Jul 06 16:38:11.135552 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:11.135518    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75577b88c-wzwhh" podStartSLOduration=0.944914995 podStartE2EDuration="8.135480956s" totalImagesPullingTime="7.190565961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:11.133736459 +0000 UTC m=+568.571247367" watchObservedRunningTime="2026-07-06 16:38:11.135480956 +0000 UTC m=+568.572991866"
Jul 06 16:38:12.048871 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:12.048838    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-kqszg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 16:38:12.049237 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:12.048899    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg" podUID="d3ed42a0-01d1-48bc-9cb0-51b150775b1c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 16:38:13.516029 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.513765    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"]
Jul 06 16:38:13.517850 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.517821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.522229 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.522068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kqq9q\""
Jul 06 16:38:13.522229 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.522079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 06 16:38:13.522229 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.522085    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 06 16:38:13.523333 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.523312    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 06 16:38:13.538484 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.538454    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"]
Jul 06 16:38:13.628841 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.628810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1e237e1-58e4-45c1-87b2-66655a18d2db-metrics-cert\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.629029 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.628866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e237e1-58e4-45c1-87b2-66655a18d2db-cert\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.629029 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.628913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d1e237e1-58e4-45c1-87b2-66655a18d2db-manager-config\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.629029 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.628950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpp56\" (UniqueName: \"kubernetes.io/projected/d1e237e1-58e4-45c1-87b2-66655a18d2db-kube-api-access-jpp56\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.729784 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.729739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d1e237e1-58e4-45c1-87b2-66655a18d2db-manager-config\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.729784 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.729792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpp56\" (UniqueName: \"kubernetes.io/projected/d1e237e1-58e4-45c1-87b2-66655a18d2db-kube-api-access-jpp56\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.730029 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.729830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1e237e1-58e4-45c1-87b2-66655a18d2db-metrics-cert\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.730029 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.729865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e237e1-58e4-45c1-87b2-66655a18d2db-cert\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.730417 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.730396    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d1e237e1-58e4-45c1-87b2-66655a18d2db-manager-config\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.732488 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.732467    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1e237e1-58e4-45c1-87b2-66655a18d2db-metrics-cert\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.732602 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.732583    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e237e1-58e4-45c1-87b2-66655a18d2db-cert\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.763859 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.763823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpp56\" (UniqueName: \"kubernetes.io/projected/d1e237e1-58e4-45c1-87b2-66655a18d2db-kube-api-access-jpp56\") pod \"lws-controller-manager-5db866ddc8-d9mwj\" (UID: \"d1e237e1-58e4-45c1-87b2-66655a18d2db\") " pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.829421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.829382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:13.987834 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:38:13.987800    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e237e1_58e4_45c1_87b2_66655a18d2db.slice/crio-41410cbd1c4aa8bd2b64d9fa532fa2fa1000c7554c960b86bc308d2fb3eeb3c0 WatchSource:0}: Error finding container 41410cbd1c4aa8bd2b64d9fa532fa2fa1000c7554c960b86bc308d2fb3eeb3c0: Status 404 returned error can't find the container with id 41410cbd1c4aa8bd2b64d9fa532fa2fa1000c7554c960b86bc308d2fb3eeb3c0
Jul 06 16:38:13.989314 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:13.989284    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"]
Jul 06 16:38:14.058224 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:14.058186    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj" event={"ID":"d1e237e1-58e4-45c1-87b2-66655a18d2db","Type":"ContainerStarted","Data":"41410cbd1c4aa8bd2b64d9fa532fa2fa1000c7554c960b86bc308d2fb3eeb3c0"}
Jul 06 16:38:15.049372 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:15.049343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:15.675577 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:15.675535    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"]
Jul 06 16:38:15.675809 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:15.675759    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg" podUID="d3ed42a0-01d1-48bc-9cb0-51b150775b1c" containerName="discovery" containerID="cri-o://347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c" gracePeriod=30
Jul 06 16:38:15.994615 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:15.994580    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:16.069142 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.069111    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3ed42a0-01d1-48bc-9cb0-51b150775b1c" containerID="347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c" exitCode=0
Jul 06 16:38:16.069617 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.069155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"
Jul 06 16:38:16.069617 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.069201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg" event={"ID":"d3ed42a0-01d1-48bc-9cb0-51b150775b1c","Type":"ContainerDied","Data":"347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c"}
Jul 06 16:38:16.069617 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.069255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kqszg" event={"ID":"d3ed42a0-01d1-48bc-9cb0-51b150775b1c","Type":"ContainerDied","Data":"d64300c5e0b46e8187e57da4123ba1e068a72fea35b99e05b2ea17e6f4465810"}
Jul 06 16:38:16.069617 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.069282    2578 scope.go:117] "RemoveContainer" containerID="347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c"
Jul 06 16:38:16.070790 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.070769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj" event={"ID":"d1e237e1-58e4-45c1-87b2-66655a18d2db","Type":"ContainerStarted","Data":"7c3cf320b5523bd5ff0bc18ef68073e660c4d0225847c126ea747d4d5252a5b2"}
Jul 06 16:38:16.070918 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.070890    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:38:16.082646 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.082620    2578 scope.go:117] "RemoveContainer" containerID="347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c"
Jul 06 16:38:16.083029 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:38:16.083004    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c\": container with ID starting with 347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c not found: ID does not exist" containerID="347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c"
Jul 06 16:38:16.083152 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.083044    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c"} err="failed to get container status \"347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c\": rpc error: code = NotFound desc = could not find container \"347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c\": container with ID starting with 347d84dfb91bf4cd19141eb74c6d3da22a813af5de98a7ddb3ec13abe7c2f69c not found: ID does not exist"
Jul 06 16:38:16.098434 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.098250    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj" podStartSLOduration=1.20601084 podStartE2EDuration="3.098231135s" totalImagesPullingTime="1.892220295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:16.097949987 +0000 UTC m=+573.535460921" watchObservedRunningTime="2026-07-06 16:38:16.098231135 +0000 UTC m=+573.535742044"
Jul 06 16:38:16.152896 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.152855    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlvms\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-kube-api-access-mlvms\") pod \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") "
Jul 06 16:38:16.153091 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.152926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-local-certs\") pod \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") "
Jul 06 16:38:16.153091 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.152956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-dns-cert\") pod \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") "
Jul 06 16:38:16.153091 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.152983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-ca-configmap\") pod \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") "
Jul 06 16:38:16.153091 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.153020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-kubeconfig\") pod \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") "
Jul 06 16:38:16.153091 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.153055    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-cacerts\") pod \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") "
Jul 06 16:38:16.153351 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.153099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-token\") pod \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\" (UID: \"d3ed42a0-01d1-48bc-9cb0-51b150775b1c\") "
Jul 06 16:38:16.154315 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.154184    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d3ed42a0-01d1-48bc-9cb0-51b150775b1c" (UID: "d3ed42a0-01d1-48bc-9cb0-51b150775b1c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:38:16.156484 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.156450    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d3ed42a0-01d1-48bc-9cb0-51b150775b1c" (UID: "d3ed42a0-01d1-48bc-9cb0-51b150775b1c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:38:16.156631 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.156590    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-local-certs" (OuterVolumeSpecName: "local-certs") pod "d3ed42a0-01d1-48bc-9cb0-51b150775b1c" (UID: "d3ed42a0-01d1-48bc-9cb0-51b150775b1c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:38:16.156631 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.156608    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-kube-api-access-mlvms" (OuterVolumeSpecName: "kube-api-access-mlvms") pod "d3ed42a0-01d1-48bc-9cb0-51b150775b1c" (UID: "d3ed42a0-01d1-48bc-9cb0-51b150775b1c"). InnerVolumeSpecName "kube-api-access-mlvms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:38:16.156811 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.156785    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3ed42a0-01d1-48bc-9cb0-51b150775b1c" (UID: "d3ed42a0-01d1-48bc-9cb0-51b150775b1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:38:16.156911 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.156872    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-cacerts" (OuterVolumeSpecName: "cacerts") pod "d3ed42a0-01d1-48bc-9cb0-51b150775b1c" (UID: "d3ed42a0-01d1-48bc-9cb0-51b150775b1c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:38:16.157111 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.157087    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d3ed42a0-01d1-48bc-9cb0-51b150775b1c" (UID: "d3ed42a0-01d1-48bc-9cb0-51b150775b1c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:38:16.254056 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.253863    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-local-certs\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:38:16.254056 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.253895    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-dns-cert\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:38:16.254056 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.253908    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-csr-ca-configmap\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:38:16.254056 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.253924    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-kubeconfig\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:38:16.254056 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.253936    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-cacerts\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:38:16.254056 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.253949    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-istio-token\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:38:16.254056 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.253962    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvms\" (UniqueName: \"kubernetes.io/projected/d3ed42a0-01d1-48bc-9cb0-51b150775b1c-kube-api-access-mlvms\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:38:16.393674 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.393645    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"]
Jul 06 16:38:16.400266 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:16.400237    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kqszg"]
Jul 06 16:38:17.113368 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:17.113288    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ed42a0-01d1-48bc-9cb0-51b150775b1c" path="/var/lib/kubelet/pods/d3ed42a0-01d1-48bc-9cb0-51b150775b1c/volumes"
Jul 06 16:38:22.050690 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:22.050665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4svsh"
Jul 06 16:38:27.078846 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:38:27.078813    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5db866ddc8-d9mwj"
Jul 06 16:40:56.613981 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.613923    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-knlmk"]
Jul 06 16:40:56.614421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.614215    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ed42a0-01d1-48bc-9cb0-51b150775b1c" containerName="discovery"
Jul 06 16:40:56.614421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.614225    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ed42a0-01d1-48bc-9cb0-51b150775b1c" containerName="discovery"
Jul 06 16:40:56.614421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.614285    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ed42a0-01d1-48bc-9cb0-51b150775b1c" containerName="discovery"
Jul 06 16:40:56.616112 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.616097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-knlmk"
Jul 06 16:40:56.618707 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.618686    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 16:40:56.619762 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.619743    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 16:40:56.619860 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.619744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kpzqj\""
Jul 06 16:40:56.625984 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.625961    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-knlmk"]
Jul 06 16:40:56.669026 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.668981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkqjr\" (UniqueName: \"kubernetes.io/projected/1f066d72-8978-40a0-895d-78698bf1de03-kube-api-access-mkqjr\") pod \"authorino-58cd955675-knlmk\" (UID: \"1f066d72-8978-40a0-895d-78698bf1de03\") " pod="kuadrant-system/authorino-58cd955675-knlmk"
Jul 06 16:40:56.769940 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.769901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqjr\" (UniqueName: \"kubernetes.io/projected/1f066d72-8978-40a0-895d-78698bf1de03-kube-api-access-mkqjr\") pod \"authorino-58cd955675-knlmk\" (UID: \"1f066d72-8978-40a0-895d-78698bf1de03\") " pod="kuadrant-system/authorino-58cd955675-knlmk"
Jul 06 16:40:56.781062 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.781028    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqjr\" (UniqueName: \"kubernetes.io/projected/1f066d72-8978-40a0-895d-78698bf1de03-kube-api-access-mkqjr\") pod \"authorino-58cd955675-knlmk\" (UID: \"1f066d72-8978-40a0-895d-78698bf1de03\") " pod="kuadrant-system/authorino-58cd955675-knlmk"
Jul 06 16:40:56.799640 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.799604    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-vdmzz"]
Jul 06 16:40:56.801959 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.801937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vdmzz"
Jul 06 16:40:56.807719 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.807693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vdmzz"]
Jul 06 16:40:56.871085 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.870991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkf7f\" (UniqueName: \"kubernetes.io/projected/88d6e1ce-3381-4095-85ac-284ae20a032a-kube-api-access-wkf7f\") pod \"authorino-75d556bd77-vdmzz\" (UID: \"88d6e1ce-3381-4095-85ac-284ae20a032a\") " pod="kuadrant-system/authorino-75d556bd77-vdmzz"
Jul 06 16:40:56.925474 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.925441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-knlmk"
Jul 06 16:40:56.972030 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.971994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkf7f\" (UniqueName: \"kubernetes.io/projected/88d6e1ce-3381-4095-85ac-284ae20a032a-kube-api-access-wkf7f\") pod \"authorino-75d556bd77-vdmzz\" (UID: \"88d6e1ce-3381-4095-85ac-284ae20a032a\") " pod="kuadrant-system/authorino-75d556bd77-vdmzz"
Jul 06 16:40:56.981211 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:56.981187    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkf7f\" (UniqueName: \"kubernetes.io/projected/88d6e1ce-3381-4095-85ac-284ae20a032a-kube-api-access-wkf7f\") pod \"authorino-75d556bd77-vdmzz\" (UID: \"88d6e1ce-3381-4095-85ac-284ae20a032a\") " pod="kuadrant-system/authorino-75d556bd77-vdmzz"
Jul 06 16:40:57.057561 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:57.057536    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-knlmk"]
Jul 06 16:40:57.060193 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:40:57.060164    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f066d72_8978_40a0_895d_78698bf1de03.slice/crio-ebb76e7b59080628a7eb4107cd32d5c42438c3ea1901ed3b70e4ca512d985041 WatchSource:0}: Error finding container ebb76e7b59080628a7eb4107cd32d5c42438c3ea1901ed3b70e4ca512d985041: Status 404 returned error can't find the container with id ebb76e7b59080628a7eb4107cd32d5c42438c3ea1901ed3b70e4ca512d985041
Jul 06 16:40:57.111982 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:57.111953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vdmzz"
Jul 06 16:40:57.242793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:57.242709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vdmzz"]
Jul 06 16:40:57.244903 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:40:57.244874    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88d6e1ce_3381_4095_85ac_284ae20a032a.slice/crio-6cf1079d319f767ed341d2fcf81a5c59eb628457d08c4488d78053e2ceb9f5fd WatchSource:0}: Error finding container 6cf1079d319f767ed341d2fcf81a5c59eb628457d08c4488d78053e2ceb9f5fd: Status 404 returned error can't find the container with id 6cf1079d319f767ed341d2fcf81a5c59eb628457d08c4488d78053e2ceb9f5fd
Jul 06 16:40:57.734316 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:57.734276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vdmzz" event={"ID":"88d6e1ce-3381-4095-85ac-284ae20a032a","Type":"ContainerStarted","Data":"6cf1079d319f767ed341d2fcf81a5c59eb628457d08c4488d78053e2ceb9f5fd"}
Jul 06 16:40:57.735396 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:40:57.735370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-knlmk" event={"ID":"1f066d72-8978-40a0-895d-78698bf1de03","Type":"ContainerStarted","Data":"ebb76e7b59080628a7eb4107cd32d5c42438c3ea1901ed3b70e4ca512d985041"}
Jul 06 16:41:01.756522 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:01.756468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vdmzz" event={"ID":"88d6e1ce-3381-4095-85ac-284ae20a032a","Type":"ContainerStarted","Data":"0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee"}
Jul 06 16:41:01.757954 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:01.757933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-knlmk" event={"ID":"1f066d72-8978-40a0-895d-78698bf1de03","Type":"ContainerStarted","Data":"c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f"}
Jul 06 16:41:01.773161 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:01.773109    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-vdmzz" podStartSLOduration=1.7455508960000001 podStartE2EDuration="5.773095675s" totalImagesPullingTime="4.027544779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:41:01.772447011 +0000 UTC m=+739.209957921" watchObservedRunningTime="2026-07-06 16:41:01.773095675 +0000 UTC m=+739.210606585"
Jul 06 16:41:01.790015 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:01.789951    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-knlmk" podStartSLOduration=1.573739966 podStartE2EDuration="5.789933479s" totalImagesPullingTime="4.216193513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:41:01.788576033 +0000 UTC m=+739.226086945" watchObservedRunningTime="2026-07-06 16:41:01.789933479 +0000 UTC m=+739.227444388"
Jul 06 16:41:01.812795 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:01.812763    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-knlmk"]
Jul 06 16:41:03.766906 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:03.766858    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-knlmk" podUID="1f066d72-8978-40a0-895d-78698bf1de03" containerName="authorino" containerID="cri-o://c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f" gracePeriod=30
Jul 06 16:41:04.008300 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.008269    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-knlmk"
Jul 06 16:41:04.030884 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.030815    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkqjr\" (UniqueName: \"kubernetes.io/projected/1f066d72-8978-40a0-895d-78698bf1de03-kube-api-access-mkqjr\") pod \"1f066d72-8978-40a0-895d-78698bf1de03\" (UID: \"1f066d72-8978-40a0-895d-78698bf1de03\") "
Jul 06 16:41:04.033166 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.033132    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f066d72-8978-40a0-895d-78698bf1de03-kube-api-access-mkqjr" (OuterVolumeSpecName: "kube-api-access-mkqjr") pod "1f066d72-8978-40a0-895d-78698bf1de03" (UID: "1f066d72-8978-40a0-895d-78698bf1de03"). InnerVolumeSpecName "kube-api-access-mkqjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:41:04.132040 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.131989    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkqjr\" (UniqueName: \"kubernetes.io/projected/1f066d72-8978-40a0-895d-78698bf1de03-kube-api-access-mkqjr\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:41:04.771739 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.771701    2578 generic.go:356] "Generic (PLEG): container finished" podID="1f066d72-8978-40a0-895d-78698bf1de03" containerID="c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f" exitCode=0
Jul 06 16:41:04.772146 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.771748    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-knlmk"
Jul 06 16:41:04.772146 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.771789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-knlmk" event={"ID":"1f066d72-8978-40a0-895d-78698bf1de03","Type":"ContainerDied","Data":"c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f"}
Jul 06 16:41:04.772146 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.771829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-knlmk" event={"ID":"1f066d72-8978-40a0-895d-78698bf1de03","Type":"ContainerDied","Data":"ebb76e7b59080628a7eb4107cd32d5c42438c3ea1901ed3b70e4ca512d985041"}
Jul 06 16:41:04.772146 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.771847    2578 scope.go:117] "RemoveContainer" containerID="c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f"
Jul 06 16:41:04.781346 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.781328    2578 scope.go:117] "RemoveContainer" containerID="c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f"
Jul 06 16:41:04.781622 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:41:04.781606    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f\": container with ID starting with c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f not found: ID does not exist" containerID="c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f"
Jul 06 16:41:04.781692 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.781631    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f"} err="failed to get container status \"c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f\": rpc error: code = NotFound desc = could not find container \"c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f\": container with ID starting with c1844d67a2d52eb04e01a83b79a8322f4981acf73ad40932437edeae64ed038f not found: ID does not exist"
Jul 06 16:41:04.795273 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.795247    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-knlmk"]
Jul 06 16:41:04.798863 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:04.798840    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-knlmk"]
Jul 06 16:41:05.111857 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:05.111823    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f066d72-8978-40a0-895d-78698bf1de03" path="/var/lib/kubelet/pods/1f066d72-8978-40a0-895d-78698bf1de03/volumes"
Jul 06 16:41:29.235488 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.235456    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w8nnf"]
Jul 06 16:41:29.235969 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.235761    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f066d72-8978-40a0-895d-78698bf1de03" containerName="authorino"
Jul 06 16:41:29.235969 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.235774    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f066d72-8978-40a0-895d-78698bf1de03" containerName="authorino"
Jul 06 16:41:29.235969 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.235824    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f066d72-8978-40a0-895d-78698bf1de03" containerName="authorino"
Jul 06 16:41:29.238737 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.238718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-w8nnf"
Jul 06 16:41:29.247223 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.247195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w8nnf"]
Jul 06 16:41:29.338959 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.338920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvftr\" (UniqueName: \"kubernetes.io/projected/00d25c39-92e0-414b-b632-3f1dc360de49-kube-api-access-zvftr\") pod \"authorino-5d4b67c446-w8nnf\" (UID: \"00d25c39-92e0-414b-b632-3f1dc360de49\") " pod="kuadrant-system/authorino-5d4b67c446-w8nnf"
Jul 06 16:41:29.439687 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.439649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvftr\" (UniqueName: \"kubernetes.io/projected/00d25c39-92e0-414b-b632-3f1dc360de49-kube-api-access-zvftr\") pod \"authorino-5d4b67c446-w8nnf\" (UID: \"00d25c39-92e0-414b-b632-3f1dc360de49\") " pod="kuadrant-system/authorino-5d4b67c446-w8nnf"
Jul 06 16:41:29.452259 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.452227    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvftr\" (UniqueName: \"kubernetes.io/projected/00d25c39-92e0-414b-b632-3f1dc360de49-kube-api-access-zvftr\") pod \"authorino-5d4b67c446-w8nnf\" (UID: \"00d25c39-92e0-414b-b632-3f1dc360de49\") " pod="kuadrant-system/authorino-5d4b67c446-w8nnf"
Jul 06 16:41:29.493981 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.493900    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w8nnf"]
Jul 06 16:41:29.494145 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.494133    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-w8nnf"
Jul 06 16:41:29.515550 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.515495    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d7bd7c54f-zhxjf"]
Jul 06 16:41:29.519049 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.519029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d7bd7c54f-zhxjf"
Jul 06 16:41:29.525993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.525965    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d7bd7c54f-zhxjf"]
Jul 06 16:41:29.544990 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.544954    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d7bd7c54f-zhxjf"]
Jul 06 16:41:29.545272 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:41:29.545247    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-c7vpz], unattached volumes=[], failed to process volumes=[kube-api-access-c7vpz]: context canceled" pod="kuadrant-system/authorino-6d7bd7c54f-zhxjf" podUID="49b73c2a-7902-4a59-bc7b-3da53e9f032c"
Jul 06 16:41:29.570651 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.570624    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-684bb46fcc-6lwwr"]
Jul 06 16:41:29.573748 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.573726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:41:29.576400 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.576376    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 06 16:41:29.581409 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.581386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-684bb46fcc-6lwwr"]
Jul 06 16:41:29.632878 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.632847    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w8nnf"]
Jul 06 16:41:29.635658 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:41:29.635633    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00d25c39_92e0_414b_b632_3f1dc360de49.slice/crio-4308f496aff2b400ef6fc4e6f2a52ae210835656a5d14b85ddfe905e7c204be8 WatchSource:0}: Error finding container 4308f496aff2b400ef6fc4e6f2a52ae210835656a5d14b85ddfe905e7c204be8: Status 404 returned error can't find the container with id 4308f496aff2b400ef6fc4e6f2a52ae210835656a5d14b85ddfe905e7c204be8
Jul 06 16:41:29.640618 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.640587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/edced3ee-e703-44c7-a127-ca6909828cef-tls-cert\") pod \"authorino-684bb46fcc-6lwwr\" (UID: \"edced3ee-e703-44c7-a127-ca6909828cef\") " pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:41:29.640698 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.640649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nh6\" (UniqueName: \"kubernetes.io/projected/edced3ee-e703-44c7-a127-ca6909828cef-kube-api-access-x2nh6\") pod \"authorino-684bb46fcc-6lwwr\" (UID: \"edced3ee-e703-44c7-a127-ca6909828cef\") " pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:41:29.640698 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.640671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vpz\" (UniqueName: \"kubernetes.io/projected/49b73c2a-7902-4a59-bc7b-3da53e9f032c-kube-api-access-c7vpz\") pod \"authorino-6d7bd7c54f-zhxjf\" (UID: \"49b73c2a-7902-4a59-bc7b-3da53e9f032c\") " pod="kuadrant-system/authorino-6d7bd7c54f-zhxjf"
Jul 06 16:41:29.742024 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.741984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/edced3ee-e703-44c7-a127-ca6909828cef-tls-cert\") pod \"authorino-684bb46fcc-6lwwr\" (UID: \"edced3ee-e703-44c7-a127-ca6909828cef\") " pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:41:29.742229 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.742036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nh6\" (UniqueName: \"kubernetes.io/projected/edced3ee-e703-44c7-a127-ca6909828cef-kube-api-access-x2nh6\") pod \"authorino-684bb46fcc-6lwwr\" (UID: \"edced3ee-e703-44c7-a127-ca6909828cef\") " pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:41:29.742229 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.742058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vpz\" (UniqueName: \"kubernetes.io/projected/49b73c2a-7902-4a59-bc7b-3da53e9f032c-kube-api-access-c7vpz\") pod \"authorino-6d7bd7c54f-zhxjf\" (UID: \"49b73c2a-7902-4a59-bc7b-3da53e9f032c\") " pod="kuadrant-system/authorino-6d7bd7c54f-zhxjf"
Jul 06 16:41:29.744664 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.744602    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/edced3ee-e703-44c7-a127-ca6909828cef-tls-cert\") pod \"authorino-684bb46fcc-6lwwr\" (UID: \"edced3ee-e703-44c7-a127-ca6909828cef\") " pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:41:29.750417 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.750391    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nh6\" (UniqueName: \"kubernetes.io/projected/edced3ee-e703-44c7-a127-ca6909828cef-kube-api-access-x2nh6\") pod \"authorino-684bb46fcc-6lwwr\" (UID: \"edced3ee-e703-44c7-a127-ca6909828cef\") " pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:41:29.750654 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.750635    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vpz\" (UniqueName: \"kubernetes.io/projected/49b73c2a-7902-4a59-bc7b-3da53e9f032c-kube-api-access-c7vpz\") pod \"authorino-6d7bd7c54f-zhxjf\" (UID: \"49b73c2a-7902-4a59-bc7b-3da53e9f032c\") " pod="kuadrant-system/authorino-6d7bd7c54f-zhxjf"
Jul 06 16:41:29.878795 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.878766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d7bd7c54f-zhxjf"
Jul 06 16:41:29.878983 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.878767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-w8nnf" event={"ID":"00d25c39-92e0-414b-b632-3f1dc360de49","Type":"ContainerStarted","Data":"4308f496aff2b400ef6fc4e6f2a52ae210835656a5d14b85ddfe905e7c204be8"}
Jul 06 16:41:29.883037 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.883008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:41:29.888476 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:29.888455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d7bd7c54f-zhxjf"
Jul 06 16:41:30.014287 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.014263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-684bb46fcc-6lwwr"]
Jul 06 16:41:30.016221 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:41:30.016192    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedced3ee_e703_44c7_a127_ca6909828cef.slice/crio-8ec3bb71f79d1d07114c3e9a17477af1af7a402d129b03b964809e585ae4b785 WatchSource:0}: Error finding container 8ec3bb71f79d1d07114c3e9a17477af1af7a402d129b03b964809e585ae4b785: Status 404 returned error can't find the container with id 8ec3bb71f79d1d07114c3e9a17477af1af7a402d129b03b964809e585ae4b785
Jul 06 16:41:30.045396 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.045368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7vpz\" (UniqueName: \"kubernetes.io/projected/49b73c2a-7902-4a59-bc7b-3da53e9f032c-kube-api-access-c7vpz\") pod \"49b73c2a-7902-4a59-bc7b-3da53e9f032c\" (UID: \"49b73c2a-7902-4a59-bc7b-3da53e9f032c\") "
Jul 06 16:41:30.047637 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.047614    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b73c2a-7902-4a59-bc7b-3da53e9f032c-kube-api-access-c7vpz" (OuterVolumeSpecName: "kube-api-access-c7vpz") pod "49b73c2a-7902-4a59-bc7b-3da53e9f032c" (UID: "49b73c2a-7902-4a59-bc7b-3da53e9f032c"). InnerVolumeSpecName "kube-api-access-c7vpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:41:30.145885 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.145862    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7vpz\" (UniqueName: \"kubernetes.io/projected/49b73c2a-7902-4a59-bc7b-3da53e9f032c-kube-api-access-c7vpz\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:41:30.884191 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.884157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-w8nnf" event={"ID":"00d25c39-92e0-414b-b632-3f1dc360de49","Type":"ContainerStarted","Data":"a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce"}
Jul 06 16:41:30.884660 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.884206    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-w8nnf" podUID="00d25c39-92e0-414b-b632-3f1dc360de49" containerName="authorino" containerID="cri-o://a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce" gracePeriod=30
Jul 06 16:41:30.885840 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.885819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-684bb46fcc-6lwwr" event={"ID":"edced3ee-e703-44c7-a127-ca6909828cef","Type":"ContainerStarted","Data":"7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02"}
Jul 06 16:41:30.885932 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.885855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-684bb46fcc-6lwwr" event={"ID":"edced3ee-e703-44c7-a127-ca6909828cef","Type":"ContainerStarted","Data":"8ec3bb71f79d1d07114c3e9a17477af1af7a402d129b03b964809e585ae4b785"}
Jul 06 16:41:30.885932 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.885879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d7bd7c54f-zhxjf"
Jul 06 16:41:30.898739 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.898699    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-w8nnf" podStartSLOduration=1.428170089 podStartE2EDuration="1.898679421s" totalImagesPullingTime="470.509332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:41:30.897996223 +0000 UTC m=+768.335507138" watchObservedRunningTime="2026-07-06 16:41:30.898679421 +0000 UTC m=+768.336190330"
Jul 06 16:41:30.917558 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.917492    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-684bb46fcc-6lwwr" podStartSLOduration=1.527604341 podStartE2EDuration="1.91747579s" totalImagesPullingTime="389.871449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:41:30.917114098 +0000 UTC m=+768.354625008" watchObservedRunningTime="2026-07-06 16:41:30.91747579 +0000 UTC m=+768.354986699"
Jul 06 16:41:30.943493 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.943461    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vdmzz"]
Jul 06 16:41:30.943717 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:30.943666    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-vdmzz" podUID="88d6e1ce-3381-4095-85ac-284ae20a032a" containerName="authorino" containerID="cri-o://0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee" gracePeriod=30
Jul 06 16:41:31.028098 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.028066    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d7bd7c54f-zhxjf"]
Jul 06 16:41:31.030108 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.030084    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d7bd7c54f-zhxjf"]
Jul 06 16:41:31.114802 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.114768    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b73c2a-7902-4a59-bc7b-3da53e9f032c" path="/var/lib/kubelet/pods/49b73c2a-7902-4a59-bc7b-3da53e9f032c/volumes"
Jul 06 16:41:31.202175 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.202151    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-w8nnf"
Jul 06 16:41:31.204806 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.204788    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vdmzz"
Jul 06 16:41:31.355192 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.355159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkf7f\" (UniqueName: \"kubernetes.io/projected/88d6e1ce-3381-4095-85ac-284ae20a032a-kube-api-access-wkf7f\") pod \"88d6e1ce-3381-4095-85ac-284ae20a032a\" (UID: \"88d6e1ce-3381-4095-85ac-284ae20a032a\") "
Jul 06 16:41:31.355377 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.355203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvftr\" (UniqueName: \"kubernetes.io/projected/00d25c39-92e0-414b-b632-3f1dc360de49-kube-api-access-zvftr\") pod \"00d25c39-92e0-414b-b632-3f1dc360de49\" (UID: \"00d25c39-92e0-414b-b632-3f1dc360de49\") "
Jul 06 16:41:31.357434 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.357406    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d25c39-92e0-414b-b632-3f1dc360de49-kube-api-access-zvftr" (OuterVolumeSpecName: "kube-api-access-zvftr") pod "00d25c39-92e0-414b-b632-3f1dc360de49" (UID: "00d25c39-92e0-414b-b632-3f1dc360de49"). InnerVolumeSpecName "kube-api-access-zvftr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:41:31.357552 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.357435    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d6e1ce-3381-4095-85ac-284ae20a032a-kube-api-access-wkf7f" (OuterVolumeSpecName: "kube-api-access-wkf7f") pod "88d6e1ce-3381-4095-85ac-284ae20a032a" (UID: "88d6e1ce-3381-4095-85ac-284ae20a032a"). InnerVolumeSpecName "kube-api-access-wkf7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:41:31.456831 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.456802    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkf7f\" (UniqueName: \"kubernetes.io/projected/88d6e1ce-3381-4095-85ac-284ae20a032a-kube-api-access-wkf7f\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:41:31.456928 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.456837    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvftr\" (UniqueName: \"kubernetes.io/projected/00d25c39-92e0-414b-b632-3f1dc360de49-kube-api-access-zvftr\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:41:31.890832 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.890800    2578 generic.go:356] "Generic (PLEG): container finished" podID="00d25c39-92e0-414b-b632-3f1dc360de49" containerID="a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce" exitCode=0
Jul 06 16:41:31.891319 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.890870    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-w8nnf"
Jul 06 16:41:31.891319 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.890890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-w8nnf" event={"ID":"00d25c39-92e0-414b-b632-3f1dc360de49","Type":"ContainerDied","Data":"a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce"}
Jul 06 16:41:31.891319 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.890941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-w8nnf" event={"ID":"00d25c39-92e0-414b-b632-3f1dc360de49","Type":"ContainerDied","Data":"4308f496aff2b400ef6fc4e6f2a52ae210835656a5d14b85ddfe905e7c204be8"}
Jul 06 16:41:31.891319 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.890966    2578 scope.go:117] "RemoveContainer" containerID="a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce"
Jul 06 16:41:31.892330 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.892046    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d6e1ce-3381-4095-85ac-284ae20a032a" containerID="0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee" exitCode=0
Jul 06 16:41:31.892330 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.892095    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vdmzz"
Jul 06 16:41:31.892330 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.892141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vdmzz" event={"ID":"88d6e1ce-3381-4095-85ac-284ae20a032a","Type":"ContainerDied","Data":"0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee"}
Jul 06 16:41:31.892330 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.892172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vdmzz" event={"ID":"88d6e1ce-3381-4095-85ac-284ae20a032a","Type":"ContainerDied","Data":"6cf1079d319f767ed341d2fcf81a5c59eb628457d08c4488d78053e2ceb9f5fd"}
Jul 06 16:41:31.900446 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.900431    2578 scope.go:117] "RemoveContainer" containerID="a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce"
Jul 06 16:41:31.900768 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:41:31.900749    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce\": container with ID starting with a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce not found: ID does not exist" containerID="a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce"
Jul 06 16:41:31.900870 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.900775    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce"} err="failed to get container status \"a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce\": rpc error: code = NotFound desc = could not find container \"a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce\": container with ID starting with a40c70e647adc26a6e4b9514453a85dec069a859c866d825e78039254ca1d8ce not found: ID does not exist"
Jul 06 16:41:31.900870 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.900792    2578 scope.go:117] "RemoveContainer" containerID="0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee"
Jul 06 16:41:31.908330 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.908311    2578 scope.go:117] "RemoveContainer" containerID="0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee"
Jul 06 16:41:31.908598 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:41:31.908581    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee\": container with ID starting with 0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee not found: ID does not exist" containerID="0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee"
Jul 06 16:41:31.908660 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.908604    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee"} err="failed to get container status \"0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee\": rpc error: code = NotFound desc = could not find container \"0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee\": container with ID starting with 0b2761ff9b501fe34d808a5d86b109f18e28faf83a8786ff4c866a20114e62ee not found: ID does not exist"
Jul 06 16:41:31.912220 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.912197    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w8nnf"]
Jul 06 16:41:31.916950 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.916927    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w8nnf"]
Jul 06 16:41:31.928486 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.928460    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vdmzz"]
Jul 06 16:41:31.931700 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:31.931679    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vdmzz"]
Jul 06 16:41:33.112471 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:33.112437    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d25c39-92e0-414b-b632-3f1dc360de49" path="/var/lib/kubelet/pods/00d25c39-92e0-414b-b632-3f1dc360de49/volumes"
Jul 06 16:41:33.112905 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:33.112777    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d6e1ce-3381-4095-85ac-284ae20a032a" path="/var/lib/kubelet/pods/88d6e1ce-3381-4095-85ac-284ae20a032a/volumes"
Jul 06 16:41:35.623007 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.622970    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75bdb5777f-gmtqm"]
Jul 06 16:41:35.623454 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.623243    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d6e1ce-3381-4095-85ac-284ae20a032a" containerName="authorino"
Jul 06 16:41:35.623454 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.623254    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d6e1ce-3381-4095-85ac-284ae20a032a" containerName="authorino"
Jul 06 16:41:35.623454 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.623271    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d25c39-92e0-414b-b632-3f1dc360de49" containerName="authorino"
Jul 06 16:41:35.623454 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.623276    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d25c39-92e0-414b-b632-3f1dc360de49" containerName="authorino"
Jul 06 16:41:35.623454 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.623320    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d25c39-92e0-414b-b632-3f1dc360de49" containerName="authorino"
Jul 06 16:41:35.623454 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.623329    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d6e1ce-3381-4095-85ac-284ae20a032a" containerName="authorino"
Jul 06 16:41:35.628221 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.628194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.630979 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.630950    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 16:41:35.630979 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.630965    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-66c55\""
Jul 06 16:41:35.631161 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.630950    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 16:41:35.639584 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.639495    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75bdb5777f-gmtqm"]
Jul 06 16:41:35.792065 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.792031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-service-ca\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.792258 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.792088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-webhook-cert\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.792258 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.792169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nqg\" (UniqueName: \"kubernetes.io/projected/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-kube-api-access-q9nqg\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.892705 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.892609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-service-ca\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.892705 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.892659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-webhook-cert\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.892705 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.892687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nqg\" (UniqueName: \"kubernetes.io/projected/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-kube-api-access-q9nqg\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.893382 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.893354    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-service-ca\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.895315 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.895290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-webhook-cert\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.901218 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.901188    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nqg\" (UniqueName: \"kubernetes.io/projected/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-kube-api-access-q9nqg\") pod \"maas-controller-75bdb5777f-gmtqm\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") " pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:35.941317 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:35.941271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:36.078119 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:36.077942    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75bdb5777f-gmtqm"]
Jul 06 16:41:36.080868 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:41:36.080837    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f6a13f_ab5d_4cee_917c_8a3058aa9148.slice/crio-3dbc72a46a3a4c0c023572f2b9478b9465fadd890a1b0ef82e451cc117d11e61 WatchSource:0}: Error finding container 3dbc72a46a3a4c0c023572f2b9478b9465fadd890a1b0ef82e451cc117d11e61: Status 404 returned error can't find the container with id 3dbc72a46a3a4c0c023572f2b9478b9465fadd890a1b0ef82e451cc117d11e61
Jul 06 16:41:36.916577 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:36.916541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bdb5777f-gmtqm" event={"ID":"f8f6a13f-ab5d-4cee-917c-8a3058aa9148","Type":"ContainerStarted","Data":"3dbc72a46a3a4c0c023572f2b9478b9465fadd890a1b0ef82e451cc117d11e61"}
Jul 06 16:41:38.928161 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:38.928125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bdb5777f-gmtqm" event={"ID":"f8f6a13f-ab5d-4cee-917c-8a3058aa9148","Type":"ContainerStarted","Data":"c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94"}
Jul 06 16:41:38.928576 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:38.928242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:41:38.958153 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:38.958043    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75bdb5777f-gmtqm" podStartSLOduration=1.357911914 podStartE2EDuration="3.958023273s" totalImagesPullingTime="2.600111359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:41:38.956282908 +0000 UTC m=+776.393793864" watchObservedRunningTime="2026-07-06 16:41:38.958023273 +0000 UTC m=+776.395534184"
Jul 06 16:41:49.938753 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:41:49.938719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:42:11.912703 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:11.912613    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-755d4b45b-drtsh"]
Jul 06 16:42:11.917767 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:11.917742    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:11.920406 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:11.920379    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 06 16:42:11.920581 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:11.920404    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jdcs4\""
Jul 06 16:42:11.928465 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:11.928443    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-755d4b45b-drtsh"]
Jul 06 16:42:11.979430 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:11.979392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfs2c\" (UniqueName: \"kubernetes.io/projected/531565ef-0ed0-4616-be99-075361340f05-kube-api-access-lfs2c\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:11.979668 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:11.979445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531565ef-0ed0-4616-be99-075361340f05-openshift-service-ca\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:11.979668 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:11.979607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531565ef-0ed0-4616-be99-075361340f05-maas-api-tls\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.080234 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.080200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531565ef-0ed0-4616-be99-075361340f05-openshift-service-ca\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.080422 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.080271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531565ef-0ed0-4616-be99-075361340f05-maas-api-tls\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.080422 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.080311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfs2c\" (UniqueName: \"kubernetes.io/projected/531565ef-0ed0-4616-be99-075361340f05-kube-api-access-lfs2c\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.080541 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:42:12.080419    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 06 16:42:12.080541 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:42:12.080482    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531565ef-0ed0-4616-be99-075361340f05-maas-api-tls podName:531565ef-0ed0-4616-be99-075361340f05 nodeName:}" failed. No retries permitted until 2026-07-06 16:42:12.580465331 +0000 UTC m=+810.017976222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/531565ef-0ed0-4616-be99-075361340f05-maas-api-tls") pod "maas-api-755d4b45b-drtsh" (UID: "531565ef-0ed0-4616-be99-075361340f05") : secret "maas-api-serving-cert" not found
Jul 06 16:42:12.080952 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.080932    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531565ef-0ed0-4616-be99-075361340f05-openshift-service-ca\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.089333 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.089303    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfs2c\" (UniqueName: \"kubernetes.io/projected/531565ef-0ed0-4616-be99-075361340f05-kube-api-access-lfs2c\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.433118 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.433078    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kr2kb"]
Jul 06 16:42:12.435880 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.435856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:12.438382 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.438361    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-57dzr\""
Jul 06 16:42:12.438382 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.438377    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 06 16:42:12.448459 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.448438    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kr2kb"]
Jul 06 16:42:12.484400 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.484373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f14186e-48f6-47dd-885a-c49c8e74e7ec-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kr2kb\" (UID: \"0f14186e-48f6-47dd-885a-c49c8e74e7ec\") " pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:12.484400 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.484404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jqw\" (UniqueName: \"kubernetes.io/projected/0f14186e-48f6-47dd-885a-c49c8e74e7ec-kube-api-access-k2jqw\") pod \"payload-pre-processing-7964457696-kr2kb\" (UID: \"0f14186e-48f6-47dd-885a-c49c8e74e7ec\") " pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:12.585581 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.585552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531565ef-0ed0-4616-be99-075361340f05-maas-api-tls\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.585767 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.585604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f14186e-48f6-47dd-885a-c49c8e74e7ec-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kr2kb\" (UID: \"0f14186e-48f6-47dd-885a-c49c8e74e7ec\") " pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:12.585767 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.585628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jqw\" (UniqueName: \"kubernetes.io/projected/0f14186e-48f6-47dd-885a-c49c8e74e7ec-kube-api-access-k2jqw\") pod \"payload-pre-processing-7964457696-kr2kb\" (UID: \"0f14186e-48f6-47dd-885a-c49c8e74e7ec\") " pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:12.586339 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.586312    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f14186e-48f6-47dd-885a-c49c8e74e7ec-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kr2kb\" (UID: \"0f14186e-48f6-47dd-885a-c49c8e74e7ec\") " pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:12.588142 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.588119    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531565ef-0ed0-4616-be99-075361340f05-maas-api-tls\") pod \"maas-api-755d4b45b-drtsh\" (UID: \"531565ef-0ed0-4616-be99-075361340f05\") " pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.593299 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.593272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jqw\" (UniqueName: \"kubernetes.io/projected/0f14186e-48f6-47dd-885a-c49c8e74e7ec-kube-api-access-k2jqw\") pod \"payload-pre-processing-7964457696-kr2kb\" (UID: \"0f14186e-48f6-47dd-885a-c49c8e74e7ec\") " pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:12.746645 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.746558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:12.828186 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.828156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:12.885033 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.884876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kr2kb"]
Jul 06 16:42:12.887423 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:42:12.887391    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f14186e_48f6_47dd_885a_c49c8e74e7ec.slice/crio-262cf6e84ff2ff3628426dbdc2ee2bea9dfa32ee772443dafbbe84ca31491ded WatchSource:0}: Error finding container 262cf6e84ff2ff3628426dbdc2ee2bea9dfa32ee772443dafbbe84ca31491ded: Status 404 returned error can't find the container with id 262cf6e84ff2ff3628426dbdc2ee2bea9dfa32ee772443dafbbe84ca31491ded
Jul 06 16:42:12.966529 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:12.966488    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-755d4b45b-drtsh"]
Jul 06 16:42:12.968904 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:42:12.968876    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531565ef_0ed0_4616_be99_075361340f05.slice/crio-3cd261d5130e941fe68e67d1eccf2c9a7297db7968e717836889d59b34d4a10c WatchSource:0}: Error finding container 3cd261d5130e941fe68e67d1eccf2c9a7297db7968e717836889d59b34d4a10c: Status 404 returned error can't find the container with id 3cd261d5130e941fe68e67d1eccf2c9a7297db7968e717836889d59b34d4a10c
Jul 06 16:42:13.076773 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:13.076729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-755d4b45b-drtsh" event={"ID":"531565ef-0ed0-4616-be99-075361340f05","Type":"ContainerStarted","Data":"3cd261d5130e941fe68e67d1eccf2c9a7297db7968e717836889d59b34d4a10c"}
Jul 06 16:42:13.077762 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:13.077737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb" event={"ID":"0f14186e-48f6-47dd-885a-c49c8e74e7ec","Type":"ContainerStarted","Data":"262cf6e84ff2ff3628426dbdc2ee2bea9dfa32ee772443dafbbe84ca31491ded"}
Jul 06 16:42:15.088828 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:15.088791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb" event={"ID":"0f14186e-48f6-47dd-885a-c49c8e74e7ec","Type":"ContainerStarted","Data":"af31675f3f473b0729832a5ba071fd59b0887c663a757641429738744b3070ae"}
Jul 06 16:42:15.089291 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:15.088862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:15.090448 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:15.090427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-755d4b45b-drtsh" event={"ID":"531565ef-0ed0-4616-be99-075361340f05","Type":"ContainerStarted","Data":"a2805ea32c84597297336e08e73bd7c2fa0a298a58631b38d8e7abf6a0a7ccb2"}
Jul 06 16:42:15.090587 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:15.090547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:15.104004 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:15.103953    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb" podStartSLOduration=1.470336791 podStartE2EDuration="3.103937261s" totalImagesPullingTime="1.63360047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:42:15.10330619 +0000 UTC m=+812.540817301" watchObservedRunningTime="2026-07-06 16:42:15.103937261 +0000 UTC m=+812.541448161"
Jul 06 16:42:15.118716 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:15.118652    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-755d4b45b-drtsh" podStartSLOduration=2.20068196 podStartE2EDuration="4.118632566s" totalImagesPullingTime="1.917950606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:42:15.118418516 +0000 UTC m=+812.555929462" watchObservedRunningTime="2026-07-06 16:42:15.118632566 +0000 UTC m=+812.556143477"
Jul 06 16:42:21.100218 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:21.100186    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-755d4b45b-drtsh"
Jul 06 16:42:26.097675 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:26.097645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-kr2kb"
Jul 06 16:42:31.397592 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.397560    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"]
Jul 06 16:42:31.400188 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.400169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.402719 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.402698    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-692dm\""
Jul 06 16:42:31.402844 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.402705    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 06 16:42:31.403790 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.403776    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 06 16:42:31.403858 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.403808    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 06 16:42:31.410403 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.410381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"]
Jul 06 16:42:31.547046 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.547002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m76b4\" (UniqueName: \"kubernetes.io/projected/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-kube-api-access-m76b4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.547046 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.547047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.547287 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.547072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.547287 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.547189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.547287 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.547237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.547287 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.547260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648414 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648414 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648414 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648727 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76b4\" (UniqueName: \"kubernetes.io/projected/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-kube-api-access-m76b4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648727 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648727 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648878 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648840    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648919 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.648919 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.648908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.650849 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.650823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.650982 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.650964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.657424 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.657400    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76b4\" (UniqueName: \"kubernetes.io/projected/efd2d63a-11ad-4c2d-bbdc-65c5d79f6912-kube-api-access-m76b4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7\" (UID: \"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.710959 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.710908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:31.863568 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.863537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"]
Jul 06 16:42:31.865923 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:42:31.865896    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd2d63a_11ad_4c2d_bbdc_65c5d79f6912.slice/crio-2ac1bbd0654a681cf4b0b2f91e9b99ad932bbf6a0ff57e124bc6d44b0851026e WatchSource:0}: Error finding container 2ac1bbd0654a681cf4b0b2f91e9b99ad932bbf6a0ff57e124bc6d44b0851026e: Status 404 returned error can't find the container with id 2ac1bbd0654a681cf4b0b2f91e9b99ad932bbf6a0ff57e124bc6d44b0851026e
Jul 06 16:42:31.867453 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:31.867434    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 16:42:32.163681 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:32.163640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7" event={"ID":"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912","Type":"ContainerStarted","Data":"2ac1bbd0654a681cf4b0b2f91e9b99ad932bbf6a0ff57e124bc6d44b0851026e"}
Jul 06 16:42:33.170156 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:33.170044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7" event={"ID":"efd2d63a-11ad-4c2d-bbdc-65c5d79f6912","Type":"ContainerStarted","Data":"b47b4d33990ec4f7c7ffe9c566d8ca89d761d3406170d726cec907af25932f06"}
Jul 06 16:42:33.188911 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:33.188857    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7" podStartSLOduration=1.140945619 podStartE2EDuration="2.188840315s" totalImagesPullingTime="1.047894696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:42:33.187721083 +0000 UTC m=+830.625231994" watchObservedRunningTime="2026-07-06 16:42:33.188840315 +0000 UTC m=+830.626351224"
Jul 06 16:42:41.711866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:41.711820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:41.712290 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:41.711900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:41.724885 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:41.724858    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:42:42.219300 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:42:42.219273    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7"
Jul 06 16:43:26.289232 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.289190    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"]
Jul 06 16:43:26.292589 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.292567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.295234 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.295210    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 06 16:43:26.301754 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.301725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"]
Jul 06 16:43:26.423495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.423457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vng\" (UniqueName: \"kubernetes.io/projected/26db21a2-dadd-467d-b367-1c0d777bbff2-kube-api-access-44vng\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.423495 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.423525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.423735 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.423564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.423735 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.423586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26db21a2-dadd-467d-b367-1c0d777bbff2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.423735 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.423677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.423832 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.423738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525145 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525145 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26db21a2-dadd-467d-b367-1c0d777bbff2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525359 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525359 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525359 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vng\" (UniqueName: \"kubernetes.io/projected/26db21a2-dadd-467d-b367-1c0d777bbff2-kube-api-access-44vng\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525491 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525703 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525679    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525808 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525692    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.525870 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.525800    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.527623 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.527605    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26db21a2-dadd-467d-b367-1c0d777bbff2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.527892 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.527872    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26db21a2-dadd-467d-b367-1c0d777bbff2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.536070 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.536044    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vng\" (UniqueName: \"kubernetes.io/projected/26db21a2-dadd-467d-b367-1c0d777bbff2-kube-api-access-44vng\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr\" (UID: \"26db21a2-dadd-467d-b367-1c0d777bbff2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.619095 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.619047    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:26.753424 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:26.753398    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"]
Jul 06 16:43:26.755986 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:43:26.755960    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26db21a2_dadd_467d_b367_1c0d777bbff2.slice/crio-cf76093b2665f14104b267b22cc7a4586cc82f480ac3278a75fa0090a7895a1f WatchSource:0}: Error finding container cf76093b2665f14104b267b22cc7a4586cc82f480ac3278a75fa0090a7895a1f: Status 404 returned error can't find the container with id cf76093b2665f14104b267b22cc7a4586cc82f480ac3278a75fa0090a7895a1f
Jul 06 16:43:27.395733 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:27.395690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr" event={"ID":"26db21a2-dadd-467d-b367-1c0d777bbff2","Type":"ContainerStarted","Data":"cf76093b2665f14104b267b22cc7a4586cc82f480ac3278a75fa0090a7895a1f"}
Jul 06 16:43:28.401694 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:28.401656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr" event={"ID":"26db21a2-dadd-467d-b367-1c0d777bbff2","Type":"ContainerStarted","Data":"ba807e547120b9be3492506bcbd7450842f01db4672d8796e1a2db107270b014"}
Jul 06 16:43:28.424899 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:28.424837    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr" podStartSLOduration=0.938331225 podStartE2EDuration="2.424816347s" totalImagesPullingTime="1.486485122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:43:28.424639981 +0000 UTC m=+885.862150940" watchObservedRunningTime="2026-07-06 16:43:28.424816347 +0000 UTC m=+885.862327257"
Jul 06 16:43:36.619824 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:36.619787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:36.620291 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:36.619844    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:36.633118 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:36.633086    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:37.452478 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:37.452447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr"
Jul 06 16:43:38.266765 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.266724    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"]
Jul 06 16:43:38.270038 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.270013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.272450 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.272426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 06 16:43:38.280325 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.280301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"]
Jul 06 16:43:38.329897 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.329860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.330094 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.329931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb73b8fb-55fb-45dc-a151-abe077b31001-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.330094 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.329959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.330094 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.329985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59c8w\" (UniqueName: \"kubernetes.io/projected/eb73b8fb-55fb-45dc-a151-abe077b31001-kube-api-access-59c8w\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.330094 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.330070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.330257 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.330121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.431459 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.431677 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.431677 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.431677 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb73b8fb-55fb-45dc-a151-abe077b31001-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.431677 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.431677 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59c8w\" (UniqueName: \"kubernetes.io/projected/eb73b8fb-55fb-45dc-a151-abe077b31001-kube-api-access-59c8w\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.431936 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431883    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.431936 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.432043 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.431999    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.433995 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.433973    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb73b8fb-55fb-45dc-a151-abe077b31001-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.434173 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.434156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb73b8fb-55fb-45dc-a151-abe077b31001-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.439206 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.439180    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59c8w\" (UniqueName: \"kubernetes.io/projected/eb73b8fb-55fb-45dc-a151-abe077b31001-kube-api-access-59c8w\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl\" (UID: \"eb73b8fb-55fb-45dc-a151-abe077b31001\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.578962 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.578926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:38.713847 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:38.713812    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"]
Jul 06 16:43:38.716108 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:43:38.716088    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb73b8fb_55fb_45dc_a151_abe077b31001.slice/crio-8d5bef719db26ededa32703e51273d7ffa56afda1f8b74d492638007e0a02e2a WatchSource:0}: Error finding container 8d5bef719db26ededa32703e51273d7ffa56afda1f8b74d492638007e0a02e2a: Status 404 returned error can't find the container with id 8d5bef719db26ededa32703e51273d7ffa56afda1f8b74d492638007e0a02e2a
Jul 06 16:43:39.451686 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:39.451648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl" event={"ID":"eb73b8fb-55fb-45dc-a151-abe077b31001","Type":"ContainerStarted","Data":"9c6568fb5af4dc04b37aa9e49f76a567a0142e73fc456ada559c4a3aa75c9826"}
Jul 06 16:43:39.452078 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:39.451693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl" event={"ID":"eb73b8fb-55fb-45dc-a151-abe077b31001","Type":"ContainerStarted","Data":"8d5bef719db26ededa32703e51273d7ffa56afda1f8b74d492638007e0a02e2a"}
Jul 06 16:43:39.471370 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:39.471311    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl" podStartSLOduration=1.290223663 podStartE2EDuration="1.471294875s" totalImagesPullingTime="181.071212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:43:39.469288039 +0000 UTC m=+896.906798949" watchObservedRunningTime="2026-07-06 16:43:39.471294875 +0000 UTC m=+896.908805783"
Jul 06 16:43:48.579148 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:48.579106    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:48.579666 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:48.579169    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:48.591604 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:48.591576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:43:49.513671 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:43:49.513640    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl"
Jul 06 16:44:25.691191 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.691155    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57c7d7f6fb-h7gqr"]
Jul 06 16:44:25.694600 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.694579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr"
Jul 06 16:44:25.701883 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.701682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c7d7f6fb-h7gqr"]
Jul 06 16:44:25.855428 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.855388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzrr\" (UniqueName: \"kubernetes.io/projected/df8357de-5489-4946-be3c-fef1ca96f057-kube-api-access-5dzrr\") pod \"authorino-57c7d7f6fb-h7gqr\" (UID: \"df8357de-5489-4946-be3c-fef1ca96f057\") " pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr"
Jul 06 16:44:25.855631 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.855437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df8357de-5489-4946-be3c-fef1ca96f057-tls-cert\") pod \"authorino-57c7d7f6fb-h7gqr\" (UID: \"df8357de-5489-4946-be3c-fef1ca96f057\") " pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr"
Jul 06 16:44:25.956653 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.956563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzrr\" (UniqueName: \"kubernetes.io/projected/df8357de-5489-4946-be3c-fef1ca96f057-kube-api-access-5dzrr\") pod \"authorino-57c7d7f6fb-h7gqr\" (UID: \"df8357de-5489-4946-be3c-fef1ca96f057\") " pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr"
Jul 06 16:44:25.956653 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.956630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df8357de-5489-4946-be3c-fef1ca96f057-tls-cert\") pod \"authorino-57c7d7f6fb-h7gqr\" (UID: \"df8357de-5489-4946-be3c-fef1ca96f057\") " pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr"
Jul 06 16:44:25.959230 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.959212    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df8357de-5489-4946-be3c-fef1ca96f057-tls-cert\") pod \"authorino-57c7d7f6fb-h7gqr\" (UID: \"df8357de-5489-4946-be3c-fef1ca96f057\") " pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr"
Jul 06 16:44:25.964338 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:25.964312    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzrr\" (UniqueName: \"kubernetes.io/projected/df8357de-5489-4946-be3c-fef1ca96f057-kube-api-access-5dzrr\") pod \"authorino-57c7d7f6fb-h7gqr\" (UID: \"df8357de-5489-4946-be3c-fef1ca96f057\") " pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr"
Jul 06 16:44:26.005366 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:26.005330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr"
Jul 06 16:44:26.138101 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:26.138079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c7d7f6fb-h7gqr"]
Jul 06 16:44:26.140338 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:44:26.140303    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf8357de_5489_4946_be3c_fef1ca96f057.slice/crio-cce721da8a1b1ee6d99179e4835de54621dde7918b630acaeb00ec58f95c4af7 WatchSource:0}: Error finding container cce721da8a1b1ee6d99179e4835de54621dde7918b630acaeb00ec58f95c4af7: Status 404 returned error can't find the container with id cce721da8a1b1ee6d99179e4835de54621dde7918b630acaeb00ec58f95c4af7
Jul 06 16:44:26.667372 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:26.667340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr" event={"ID":"df8357de-5489-4946-be3c-fef1ca96f057","Type":"ContainerStarted","Data":"cce721da8a1b1ee6d99179e4835de54621dde7918b630acaeb00ec58f95c4af7"}
Jul 06 16:44:27.673966 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:27.673926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr" event={"ID":"df8357de-5489-4946-be3c-fef1ca96f057","Type":"ContainerStarted","Data":"9f13ee342692fe3a0ea0ab8646100044458309d13643ca3950ad244495cdf362"}
Jul 06 16:44:27.689701 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:27.689617    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57c7d7f6fb-h7gqr" podStartSLOduration=2.220067285 podStartE2EDuration="2.689594883s" totalImagesPullingTime="469.527598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:44:27.689546643 +0000 UTC m=+945.127057556" watchObservedRunningTime="2026-07-06 16:44:27.689594883 +0000 UTC m=+945.127105792"
Jul 06 16:44:27.717571 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:27.717536    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-684bb46fcc-6lwwr"]
Jul 06 16:44:27.717749 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:27.717726    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-684bb46fcc-6lwwr" podUID="edced3ee-e703-44c7-a127-ca6909828cef" containerName="authorino" containerID="cri-o://7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02" gracePeriod=30
Jul 06 16:44:27.982387 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:27.982360    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:44:28.077102 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.077066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/edced3ee-e703-44c7-a127-ca6909828cef-tls-cert\") pod \"edced3ee-e703-44c7-a127-ca6909828cef\" (UID: \"edced3ee-e703-44c7-a127-ca6909828cef\") "
Jul 06 16:44:28.077261 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.077176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2nh6\" (UniqueName: \"kubernetes.io/projected/edced3ee-e703-44c7-a127-ca6909828cef-kube-api-access-x2nh6\") pod \"edced3ee-e703-44c7-a127-ca6909828cef\" (UID: \"edced3ee-e703-44c7-a127-ca6909828cef\") "
Jul 06 16:44:28.079290 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.079259    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edced3ee-e703-44c7-a127-ca6909828cef-kube-api-access-x2nh6" (OuterVolumeSpecName: "kube-api-access-x2nh6") pod "edced3ee-e703-44c7-a127-ca6909828cef" (UID: "edced3ee-e703-44c7-a127-ca6909828cef"). InnerVolumeSpecName "kube-api-access-x2nh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:44:28.088183 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.088151    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edced3ee-e703-44c7-a127-ca6909828cef-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "edced3ee-e703-44c7-a127-ca6909828cef" (UID: "edced3ee-e703-44c7-a127-ca6909828cef"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:44:28.178616 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.178585    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nh6\" (UniqueName: \"kubernetes.io/projected/edced3ee-e703-44c7-a127-ca6909828cef-kube-api-access-x2nh6\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:44:28.178616 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.178610    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/edced3ee-e703-44c7-a127-ca6909828cef-tls-cert\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:44:28.685551 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.685481    2578 generic.go:356] "Generic (PLEG): container finished" podID="edced3ee-e703-44c7-a127-ca6909828cef" containerID="7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02" exitCode=0
Jul 06 16:44:28.686008 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.685556    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-684bb46fcc-6lwwr"
Jul 06 16:44:28.686008 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.685738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-684bb46fcc-6lwwr" event={"ID":"edced3ee-e703-44c7-a127-ca6909828cef","Type":"ContainerDied","Data":"7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02"}
Jul 06 16:44:28.686008 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.685824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-684bb46fcc-6lwwr" event={"ID":"edced3ee-e703-44c7-a127-ca6909828cef","Type":"ContainerDied","Data":"8ec3bb71f79d1d07114c3e9a17477af1af7a402d129b03b964809e585ae4b785"}
Jul 06 16:44:28.686008 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.685862    2578 scope.go:117] "RemoveContainer" containerID="7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02"
Jul 06 16:44:28.694967 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.694947    2578 scope.go:117] "RemoveContainer" containerID="7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02"
Jul 06 16:44:28.695211 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:44:28.695193    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02\": container with ID starting with 7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02 not found: ID does not exist" containerID="7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02"
Jul 06 16:44:28.695277 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.695218    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02"} err="failed to get container status \"7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02\": rpc error: code = NotFound desc = could not find container \"7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02\": container with ID starting with 7a50c4f11ec8525fd8023b95fe09a74f1e5242e09e33f2225a5e8d8be41b4e02 not found: ID does not exist"
Jul 06 16:44:28.706813 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.706787    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-684bb46fcc-6lwwr"]
Jul 06 16:44:28.709103 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:28.709079    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-684bb46fcc-6lwwr"]
Jul 06 16:44:29.111952 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:44:29.111921    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edced3ee-e703-44c7-a127-ca6909828cef" path="/var/lib/kubelet/pods/edced3ee-e703-44c7-a127-ca6909828cef/volumes"
Jul 06 16:45:43.273287 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.273256    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75bdb5777f-gmtqm"]
Jul 06 16:45:43.273774 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.273518    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-75bdb5777f-gmtqm" podUID="f8f6a13f-ab5d-4cee-917c-8a3058aa9148" containerName="manager" containerID="cri-o://c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94" gracePeriod=30
Jul 06 16:45:43.521743 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.521710    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:45:43.660836 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.660806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-service-ca\") pod \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") "
Jul 06 16:45:43.660836 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.660854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9nqg\" (UniqueName: \"kubernetes.io/projected/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-kube-api-access-q9nqg\") pod \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") "
Jul 06 16:45:43.661058 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.660886    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-webhook-cert\") pod \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\" (UID: \"f8f6a13f-ab5d-4cee-917c-8a3058aa9148\") "
Jul 06 16:45:43.661270 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.661245    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-service-ca" (OuterVolumeSpecName: "service-ca") pod "f8f6a13f-ab5d-4cee-917c-8a3058aa9148" (UID: "f8f6a13f-ab5d-4cee-917c-8a3058aa9148"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:45:43.663277 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.663247    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f8f6a13f-ab5d-4cee-917c-8a3058aa9148" (UID: "f8f6a13f-ab5d-4cee-917c-8a3058aa9148"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:45:43.663401 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.663297    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-kube-api-access-q9nqg" (OuterVolumeSpecName: "kube-api-access-q9nqg") pod "f8f6a13f-ab5d-4cee-917c-8a3058aa9148" (UID: "f8f6a13f-ab5d-4cee-917c-8a3058aa9148"). InnerVolumeSpecName "kube-api-access-q9nqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:45:43.761922 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.761880    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-webhook-cert\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:45:43.761922 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.761912    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:45:43.761922 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:43.761922    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9nqg\" (UniqueName: \"kubernetes.io/projected/f8f6a13f-ab5d-4cee-917c-8a3058aa9148-kube-api-access-q9nqg\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:45:44.010551 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.010433    2578 generic.go:356] "Generic (PLEG): container finished" podID="f8f6a13f-ab5d-4cee-917c-8a3058aa9148" containerID="c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94" exitCode=0
Jul 06 16:45:44.010551 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.010480    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bdb5777f-gmtqm"
Jul 06 16:45:44.010551 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.010541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bdb5777f-gmtqm" event={"ID":"f8f6a13f-ab5d-4cee-917c-8a3058aa9148","Type":"ContainerDied","Data":"c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94"}
Jul 06 16:45:44.010861 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.010596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bdb5777f-gmtqm" event={"ID":"f8f6a13f-ab5d-4cee-917c-8a3058aa9148","Type":"ContainerDied","Data":"3dbc72a46a3a4c0c023572f2b9478b9465fadd890a1b0ef82e451cc117d11e61"}
Jul 06 16:45:44.010861 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.010621    2578 scope.go:117] "RemoveContainer" containerID="c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94"
Jul 06 16:45:44.019580 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.019560    2578 scope.go:117] "RemoveContainer" containerID="c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94"
Jul 06 16:45:44.019846 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:45:44.019825    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94\": container with ID starting with c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94 not found: ID does not exist" containerID="c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94"
Jul 06 16:45:44.019897 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.019855    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94"} err="failed to get container status \"c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94\": rpc error: code = NotFound desc = could not find container \"c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94\": container with ID starting with c3229a2f227e88e966979b2b3df586ebbf675f51d55ce64afeb6ff07cc70ad94 not found: ID does not exist"
Jul 06 16:45:44.031908 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.031884    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75bdb5777f-gmtqm"]
Jul 06 16:45:44.034912 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:44.034892    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75bdb5777f-gmtqm"]
Jul 06 16:45:45.112111 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:45:45.112074    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f6a13f-ab5d-4cee-917c-8a3058aa9148" path="/var/lib/kubelet/pods/f8f6a13f-ab5d-4cee-917c-8a3058aa9148/volumes"
Jul 06 16:58:18.307658 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.307618    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"]
Jul 06 16:58:18.308060 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.307929    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f6a13f-ab5d-4cee-917c-8a3058aa9148" containerName="manager"
Jul 06 16:58:18.308060 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.307941    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f6a13f-ab5d-4cee-917c-8a3058aa9148" containerName="manager"
Jul 06 16:58:18.308060 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.307957    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edced3ee-e703-44c7-a127-ca6909828cef" containerName="authorino"
Jul 06 16:58:18.308060 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.307962    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="edced3ee-e703-44c7-a127-ca6909828cef" containerName="authorino"
Jul 06 16:58:18.308060 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.308017    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f6a13f-ab5d-4cee-917c-8a3058aa9148" containerName="manager"
Jul 06 16:58:18.308060 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.308025    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="edced3ee-e703-44c7-a127-ca6909828cef" containerName="authorino"
Jul 06 16:58:18.310953 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.310930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:18.313388 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.313365    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-3566469b-serving-cert\""
Jul 06 16:58:18.321149 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.321130    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"]
Jul 06 16:58:18.411809 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.411756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:18.411809 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.411818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-openshift-service-ca\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:18.412054 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.411883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztfrj\" (UniqueName: \"kubernetes.io/projected/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-kube-api-access-ztfrj\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:18.512624 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.512589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:18.512624 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.512629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-openshift-service-ca\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:18.512941 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.512659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfrj\" (UniqueName: \"kubernetes.io/projected/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-kube-api-access-ztfrj\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:18.512941 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:18.512770    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-3566469b-serving-cert: secret "maas-api-e2e-ait-3566469b-serving-cert" not found
Jul 06 16:58:18.512941 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:18.512864    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls podName:f44ac759-c0f7-45e1-8a6e-5b6338d4ec67 nodeName:}" failed. No retries permitted until 2026-07-06 16:58:19.012842716 +0000 UTC m=+1776.450353617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls") pod "maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr" (UID: "f44ac759-c0f7-45e1-8a6e-5b6338d4ec67") : secret "maas-api-e2e-ait-3566469b-serving-cert" not found
Jul 06 16:58:18.513211 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.513192    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-openshift-service-ca\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:18.522743 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:18.522720    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfrj\" (UniqueName: \"kubernetes.io/projected/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-kube-api-access-ztfrj\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:19.017824 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:19.017792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:19.020407 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:19.020383    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls\") pod \"maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") " pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:19.223071 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:19.223039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:19.353659 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:19.353579    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"]
Jul 06 16:58:19.356343 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:58:19.356305    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44ac759_c0f7_45e1_8a6e_5b6338d4ec67.slice/crio-16482b32a84debf3ac1b328ccc0ecca9e0b587c17fed802c5a5ffc0cdcd2101e WatchSource:0}: Error finding container 16482b32a84debf3ac1b328ccc0ecca9e0b587c17fed802c5a5ffc0cdcd2101e: Status 404 returned error can't find the container with id 16482b32a84debf3ac1b328ccc0ecca9e0b587c17fed802c5a5ffc0cdcd2101e
Jul 06 16:58:19.357728 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:19.357711    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 16:58:20.258431 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:20.258351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr" event={"ID":"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67","Type":"ContainerStarted","Data":"27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f"}
Jul 06 16:58:20.258431 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:20.258399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr" event={"ID":"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67","Type":"ContainerStarted","Data":"16482b32a84debf3ac1b328ccc0ecca9e0b587c17fed802c5a5ffc0cdcd2101e"}
Jul 06 16:58:20.258676 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:20.258484    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:20.275324 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:20.275272    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr" podStartSLOduration=1.7881734310000001 podStartE2EDuration="2.275256811s" totalImagesPullingTime="487.08338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:58:20.273142708 +0000 UTC m=+1777.710653637" watchObservedRunningTime="2026-07-06 16:58:20.275256811 +0000 UTC m=+1777.712767721"
Jul 06 16:58:26.220864 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.220830    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"]
Jul 06 16:58:26.221236 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.221004    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr" podUID="f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" containerName="maas-api" containerID="cri-o://27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f" gracePeriod=30
Jul 06 16:58:26.226207 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.226183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:26.474925 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.474869    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:26.572082 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.572046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-openshift-service-ca\") pod \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") "
Jul 06 16:58:26.572255 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.572100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztfrj\" (UniqueName: \"kubernetes.io/projected/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-kube-api-access-ztfrj\") pod \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") "
Jul 06 16:58:26.572255 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.572225    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls\") pod \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\" (UID: \"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67\") "
Jul 06 16:58:26.572457 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.572436    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" (UID: "f44ac759-c0f7-45e1-8a6e-5b6338d4ec67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:58:26.572565 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.572550    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:26.574313 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.574290    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-kube-api-access-ztfrj" (OuterVolumeSpecName: "kube-api-access-ztfrj") pod "f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" (UID: "f44ac759-c0f7-45e1-8a6e-5b6338d4ec67"). InnerVolumeSpecName "kube-api-access-ztfrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:58:26.574397 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.574379    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" (UID: "f44ac759-c0f7-45e1-8a6e-5b6338d4ec67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:58:26.673418 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.673386    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:26.673418 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:26.673412    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztfrj\" (UniqueName: \"kubernetes.io/projected/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67-kube-api-access-ztfrj\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:27.290477 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.290380    2578 generic.go:356] "Generic (PLEG): container finished" podID="f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" containerID="27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f" exitCode=0
Jul 06 16:58:27.290477 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.290432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr" event={"ID":"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67","Type":"ContainerDied","Data":"27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f"}
Jul 06 16:58:27.290477 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.290448    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"
Jul 06 16:58:27.291079 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.290484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr" event={"ID":"f44ac759-c0f7-45e1-8a6e-5b6338d4ec67","Type":"ContainerDied","Data":"16482b32a84debf3ac1b328ccc0ecca9e0b587c17fed802c5a5ffc0cdcd2101e"}
Jul 06 16:58:27.291079 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.290528    2578 scope.go:117] "RemoveContainer" containerID="27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f"
Jul 06 16:58:27.298943 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.298924    2578 scope.go:117] "RemoveContainer" containerID="27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f"
Jul 06 16:58:27.299179 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:27.299160    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f\": container with ID starting with 27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f not found: ID does not exist" containerID="27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f"
Jul 06 16:58:27.299248 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.299190    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f"} err="failed to get container status \"27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f\": rpc error: code = NotFound desc = could not find container \"27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f\": container with ID starting with 27d77d5951329b0d3eb04d526fa3ad696714425de82022beabf4f8371bb67e8f not found: ID does not exist"
Jul 06 16:58:27.308215 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.308194    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"]
Jul 06 16:58:27.311161 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:27.311141    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-3566469b-77755bfbf8-vkfrr"]
Jul 06 16:58:29.112050 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:29.112017    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" path="/var/lib/kubelet/pods/f44ac759-c0f7-45e1-8a6e-5b6338d4ec67/volumes"
Jul 06 16:58:33.904588 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.904558    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"]
Jul 06 16:58:33.904964 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.904926    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" containerName="maas-api"
Jul 06 16:58:33.904964 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.904940    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" containerName="maas-api"
Jul 06 16:58:33.905039 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.905020    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f44ac759-c0f7-45e1-8a6e-5b6338d4ec67" containerName="maas-api"
Jul 06 16:58:33.909906 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.909881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:33.912741 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.912714    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-b2be3c64-serving-cert\""
Jul 06 16:58:33.920993 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.920967    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"]
Jul 06 16:58:33.939633 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.939591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:33.939806 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.939645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8k5b\" (UniqueName: \"kubernetes.io/projected/205572bb-8f35-4e2e-a6ca-432b96e7c913-kube-api-access-s8k5b\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:33.939849 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:33.939801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/205572bb-8f35-4e2e-a6ca-432b96e7c913-openshift-service-ca\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.040347 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.040311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/205572bb-8f35-4e2e-a6ca-432b96e7c913-openshift-service-ca\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.040545 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.040370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.040545 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:34.040486    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-b2be3c64-serving-cert: secret "maas-api-e2e-ait-b2be3c64-serving-cert" not found
Jul 06 16:58:34.040679 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:34.040587    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls podName:205572bb-8f35-4e2e-a6ca-432b96e7c913 nodeName:}" failed. No retries permitted until 2026-07-06 16:58:34.540564599 +0000 UTC m=+1791.978075500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls") pod "maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw" (UID: "205572bb-8f35-4e2e-a6ca-432b96e7c913") : secret "maas-api-e2e-ait-b2be3c64-serving-cert" not found
Jul 06 16:58:34.040679 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.040572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8k5b\" (UniqueName: \"kubernetes.io/projected/205572bb-8f35-4e2e-a6ca-432b96e7c913-kube-api-access-s8k5b\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.041061 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.041043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/205572bb-8f35-4e2e-a6ca-432b96e7c913-openshift-service-ca\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.055717 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.055678    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8k5b\" (UniqueName: \"kubernetes.io/projected/205572bb-8f35-4e2e-a6ca-432b96e7c913-kube-api-access-s8k5b\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.543557 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.543484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.546124 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.546096    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls\") pod \"maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") " pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.822144 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.822112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:34.961991 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:34.961959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"]
Jul 06 16:58:34.964659 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:58:34.964629    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205572bb_8f35_4e2e_a6ca_432b96e7c913.slice/crio-ed9838c8b547ac68b8fe2189e81093643a57175ce01dd296df49eb6e58524485 WatchSource:0}: Error finding container ed9838c8b547ac68b8fe2189e81093643a57175ce01dd296df49eb6e58524485: Status 404 returned error can't find the container with id ed9838c8b547ac68b8fe2189e81093643a57175ce01dd296df49eb6e58524485
Jul 06 16:58:35.327225 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:35.327195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw" event={"ID":"205572bb-8f35-4e2e-a6ca-432b96e7c913","Type":"ContainerStarted","Data":"ed9838c8b547ac68b8fe2189e81093643a57175ce01dd296df49eb6e58524485"}
Jul 06 16:58:36.333333 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:36.333294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw" event={"ID":"205572bb-8f35-4e2e-a6ca-432b96e7c913","Type":"ContainerStarted","Data":"8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5"}
Jul 06 16:58:36.333850 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:36.333420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:36.355866 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:36.355811    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw" podStartSLOduration=3.025041774 podStartE2EDuration="3.355793502s" totalImagesPullingTime="330.751728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:58:36.354779209 +0000 UTC m=+1793.792290132" watchObservedRunningTime="2026-07-06 16:58:36.355793502 +0000 UTC m=+1793.793304410"
Jul 06 16:58:41.962087 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:41.962051    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"]
Jul 06 16:58:41.962494 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:41.962302    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw" podUID="205572bb-8f35-4e2e-a6ca-432b96e7c913" containerName="maas-api" containerID="cri-o://8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5" gracePeriod=30
Jul 06 16:58:41.967576 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:41.967555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:42.207259 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.207231    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:42.323640 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.323607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls\") pod \"205572bb-8f35-4e2e-a6ca-432b96e7c913\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") "
Jul 06 16:58:42.323802 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.323656    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8k5b\" (UniqueName: \"kubernetes.io/projected/205572bb-8f35-4e2e-a6ca-432b96e7c913-kube-api-access-s8k5b\") pod \"205572bb-8f35-4e2e-a6ca-432b96e7c913\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") "
Jul 06 16:58:42.323802 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.323679    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/205572bb-8f35-4e2e-a6ca-432b96e7c913-openshift-service-ca\") pod \"205572bb-8f35-4e2e-a6ca-432b96e7c913\" (UID: \"205572bb-8f35-4e2e-a6ca-432b96e7c913\") "
Jul 06 16:58:42.324114 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.324085    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205572bb-8f35-4e2e-a6ca-432b96e7c913-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "205572bb-8f35-4e2e-a6ca-432b96e7c913" (UID: "205572bb-8f35-4e2e-a6ca-432b96e7c913"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:58:42.325895 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.325870    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205572bb-8f35-4e2e-a6ca-432b96e7c913-kube-api-access-s8k5b" (OuterVolumeSpecName: "kube-api-access-s8k5b") pod "205572bb-8f35-4e2e-a6ca-432b96e7c913" (UID: "205572bb-8f35-4e2e-a6ca-432b96e7c913"). InnerVolumeSpecName "kube-api-access-s8k5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:58:42.325895 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.325873    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "205572bb-8f35-4e2e-a6ca-432b96e7c913" (UID: "205572bb-8f35-4e2e-a6ca-432b96e7c913"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:58:42.360259 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.360230    2578 generic.go:356] "Generic (PLEG): container finished" podID="205572bb-8f35-4e2e-a6ca-432b96e7c913" containerID="8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5" exitCode=0
Jul 06 16:58:42.360421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.360282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw" event={"ID":"205572bb-8f35-4e2e-a6ca-432b96e7c913","Type":"ContainerDied","Data":"8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5"}
Jul 06 16:58:42.360421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.360293    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"
Jul 06 16:58:42.360421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.360326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw" event={"ID":"205572bb-8f35-4e2e-a6ca-432b96e7c913","Type":"ContainerDied","Data":"ed9838c8b547ac68b8fe2189e81093643a57175ce01dd296df49eb6e58524485"}
Jul 06 16:58:42.360421 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.360351    2578 scope.go:117] "RemoveContainer" containerID="8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5"
Jul 06 16:58:42.369637 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.369619    2578 scope.go:117] "RemoveContainer" containerID="8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5"
Jul 06 16:58:42.369906 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:42.369888    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5\": container with ID starting with 8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5 not found: ID does not exist" containerID="8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5"
Jul 06 16:58:42.369961 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.369917    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5"} err="failed to get container status \"8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5\": rpc error: code = NotFound desc = could not find container \"8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5\": container with ID starting with 8e0b342e818e682d33aec66b7b9af96706a1114729901e9eb6ba13c786e861e5 not found: ID does not exist"
Jul 06 16:58:42.381607 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.381576    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"]
Jul 06 16:58:42.385105 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.385085    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-b2be3c64-7bd587db58-s46fw"]
Jul 06 16:58:42.425025 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.424991    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/205572bb-8f35-4e2e-a6ca-432b96e7c913-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:42.425025 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.425022    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8k5b\" (UniqueName: \"kubernetes.io/projected/205572bb-8f35-4e2e-a6ca-432b96e7c913-kube-api-access-s8k5b\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:42.425185 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:42.425034    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/205572bb-8f35-4e2e-a6ca-432b96e7c913-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:43.113584 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:43.113551    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="205572bb-8f35-4e2e-a6ca-432b96e7c913" path="/var/lib/kubelet/pods/205572bb-8f35-4e2e-a6ca-432b96e7c913/volumes"
Jul 06 16:58:50.763391 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.763359    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"]
Jul 06 16:58:50.763812 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.763707    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="205572bb-8f35-4e2e-a6ca-432b96e7c913" containerName="maas-api"
Jul 06 16:58:50.763812 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.763719    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="205572bb-8f35-4e2e-a6ca-432b96e7c913" containerName="maas-api"
Jul 06 16:58:50.763812 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.763785    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="205572bb-8f35-4e2e-a6ca-432b96e7c913" containerName="maas-api"
Jul 06 16:58:50.767957 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.767937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:50.771755 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.771733    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-9cf2069a-serving-cert\""
Jul 06 16:58:50.779956 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.779934    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"]
Jul 06 16:58:50.898892 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.898858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wz6\" (UniqueName: \"kubernetes.io/projected/77d0548d-f788-41f5-bc6c-b61fba6a8940-kube-api-access-96wz6\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:50.899071 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.898915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0548d-f788-41f5-bc6c-b61fba6a8940-openshift-service-ca\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:50.899071 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.898979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.000005 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.999916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0548d-f788-41f5-bc6c-b61fba6a8940-openshift-service-ca\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.000005 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:50.999961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.000187 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:51.000098    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-9cf2069a-serving-cert: secret "maas-api-e2e-derive-9cf2069a-serving-cert" not found
Jul 06 16:58:51.000187 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:51.000099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wz6\" (UniqueName: \"kubernetes.io/projected/77d0548d-f788-41f5-bc6c-b61fba6a8940-kube-api-access-96wz6\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.000187 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:51.000180    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls podName:77d0548d-f788-41f5-bc6c-b61fba6a8940 nodeName:}" failed. No retries permitted until 2026-07-06 16:58:51.500154683 +0000 UTC m=+1808.937665592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls") pod "maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr" (UID: "77d0548d-f788-41f5-bc6c-b61fba6a8940") : secret "maas-api-e2e-derive-9cf2069a-serving-cert" not found
Jul 06 16:58:51.000710 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:51.000692    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0548d-f788-41f5-bc6c-b61fba6a8940-openshift-service-ca\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.009011 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:51.008989    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wz6\" (UniqueName: \"kubernetes.io/projected/77d0548d-f788-41f5-bc6c-b61fba6a8940-kube-api-access-96wz6\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.503201 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:51.503166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.505800 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:51.505776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls\") pod \"maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") " pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.678054 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:51.678014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:51.813639 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:51.813473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"]
Jul 06 16:58:51.815752 ip-10-0-142-118 kubenswrapper[2578]: W0706 16:58:51.815727    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d0548d_f788_41f5_bc6c_b61fba6a8940.slice/crio-c853c60ae7f7ac5facfac465c09eadae1e6ed7c16ab32e4538c479f306b30fdf WatchSource:0}: Error finding container c853c60ae7f7ac5facfac465c09eadae1e6ed7c16ab32e4538c479f306b30fdf: Status 404 returned error can't find the container with id c853c60ae7f7ac5facfac465c09eadae1e6ed7c16ab32e4538c479f306b30fdf
Jul 06 16:58:52.409745 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:52.409709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr" event={"ID":"77d0548d-f788-41f5-bc6c-b61fba6a8940","Type":"ContainerStarted","Data":"dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9"}
Jul 06 16:58:52.409947 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:52.409763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr" event={"ID":"77d0548d-f788-41f5-bc6c-b61fba6a8940","Type":"ContainerStarted","Data":"c853c60ae7f7ac5facfac465c09eadae1e6ed7c16ab32e4538c479f306b30fdf"}
Jul 06 16:58:52.409947 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:52.409800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:52.427458 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:52.427364    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr" podStartSLOduration=2.082333491 podStartE2EDuration="2.427346389s" totalImagesPullingTime="345.012898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:58:52.425806541 +0000 UTC m=+1809.863317670" watchObservedRunningTime="2026-07-06 16:58:52.427346389 +0000 UTC m=+1809.864857297"
Jul 06 16:58:58.118518 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.118472    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"]
Jul 06 16:58:58.118916 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.118698    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr" podUID="77d0548d-f788-41f5-bc6c-b61fba6a8940" containerName="maas-api" containerID="cri-o://dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9" gracePeriod=30
Jul 06 16:58:58.125118 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.125096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:58.372247 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.372179    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:58.437616 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.437583    2578 generic.go:356] "Generic (PLEG): container finished" podID="77d0548d-f788-41f5-bc6c-b61fba6a8940" containerID="dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9" exitCode=0
Jul 06 16:58:58.437793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.437627    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"
Jul 06 16:58:58.437793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.437678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr" event={"ID":"77d0548d-f788-41f5-bc6c-b61fba6a8940","Type":"ContainerDied","Data":"dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9"}
Jul 06 16:58:58.437793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.437725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr" event={"ID":"77d0548d-f788-41f5-bc6c-b61fba6a8940","Type":"ContainerDied","Data":"c853c60ae7f7ac5facfac465c09eadae1e6ed7c16ab32e4538c479f306b30fdf"}
Jul 06 16:58:58.437793 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.437747    2578 scope.go:117] "RemoveContainer" containerID="dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9"
Jul 06 16:58:58.446136 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.446115    2578 scope.go:117] "RemoveContainer" containerID="dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9"
Jul 06 16:58:58.446419 ip-10-0-142-118 kubenswrapper[2578]: E0706 16:58:58.446402    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9\": container with ID starting with dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9 not found: ID does not exist" containerID="dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9"
Jul 06 16:58:58.446465 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.446431    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9"} err="failed to get container status \"dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9\": rpc error: code = NotFound desc = could not find container \"dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9\": container with ID starting with dbbc461eabf0b41e795ee2a406571f5927e4fa4e5f98882cc325b74b03e1e2a9 not found: ID does not exist"
Jul 06 16:58:58.472752 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.472723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0548d-f788-41f5-bc6c-b61fba6a8940-openshift-service-ca\") pod \"77d0548d-f788-41f5-bc6c-b61fba6a8940\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") "
Jul 06 16:58:58.472872 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.472778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls\") pod \"77d0548d-f788-41f5-bc6c-b61fba6a8940\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") "
Jul 06 16:58:58.472872 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.472810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96wz6\" (UniqueName: \"kubernetes.io/projected/77d0548d-f788-41f5-bc6c-b61fba6a8940-kube-api-access-96wz6\") pod \"77d0548d-f788-41f5-bc6c-b61fba6a8940\" (UID: \"77d0548d-f788-41f5-bc6c-b61fba6a8940\") "
Jul 06 16:58:58.473094 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.473074    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d0548d-f788-41f5-bc6c-b61fba6a8940-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77d0548d-f788-41f5-bc6c-b61fba6a8940" (UID: "77d0548d-f788-41f5-bc6c-b61fba6a8940"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:58:58.475084 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.475062    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d0548d-f788-41f5-bc6c-b61fba6a8940-kube-api-access-96wz6" (OuterVolumeSpecName: "kube-api-access-96wz6") pod "77d0548d-f788-41f5-bc6c-b61fba6a8940" (UID: "77d0548d-f788-41f5-bc6c-b61fba6a8940"). InnerVolumeSpecName "kube-api-access-96wz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:58:58.475084 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.475071    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77d0548d-f788-41f5-bc6c-b61fba6a8940" (UID: "77d0548d-f788-41f5-bc6c-b61fba6a8940"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:58:58.573415 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.573383    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96wz6\" (UniqueName: \"kubernetes.io/projected/77d0548d-f788-41f5-bc6c-b61fba6a8940-kube-api-access-96wz6\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:58.573415 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.573411    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0548d-f788-41f5-bc6c-b61fba6a8940-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:58.573415 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.573420    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d0548d-f788-41f5-bc6c-b61fba6a8940-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 16:58:58.759602 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.759573    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"]
Jul 06 16:58:58.763472 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:58.763446    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-9cf2069a-5d9889f864-cfdpr"]
Jul 06 16:58:59.112604 ip-10-0-142-118 kubenswrapper[2578]: I0706 16:58:59.112574    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d0548d-f788-41f5-bc6c-b61fba6a8940" path="/var/lib/kubelet/pods/77d0548d-f788-41f5-bc6c-b61fba6a8940/volumes"
Jul 06 17:00:43.753864 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.753830    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"]
Jul 06 17:00:43.754280 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.754226    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d0548d-f788-41f5-bc6c-b61fba6a8940" containerName="maas-api"
Jul 06 17:00:43.754280 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.754238    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d0548d-f788-41f5-bc6c-b61fba6a8940" containerName="maas-api"
Jul 06 17:00:43.754353 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.754296    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d0548d-f788-41f5-bc6c-b61fba6a8940" containerName="maas-api"
Jul 06 17:00:43.757180 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.757163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:43.760142 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.760118    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-9395f82e-serving-cert\""
Jul 06 17:00:43.787846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.787816    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"]
Jul 06 17:00:43.805987 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.805939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlfk\" (UniqueName: \"kubernetes.io/projected/dfe06055-49aa-4680-bc0c-a53b25f21552-kube-api-access-mtlfk\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:43.806150 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.805995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe06055-49aa-4680-bc0c-a53b25f21552-openshift-service-ca\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:43.806150 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.806059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:43.906775 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.906741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlfk\" (UniqueName: \"kubernetes.io/projected/dfe06055-49aa-4680-bc0c-a53b25f21552-kube-api-access-mtlfk\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:43.906943 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.906778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe06055-49aa-4680-bc0c-a53b25f21552-openshift-service-ca\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:43.906943 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.906844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:43.907060 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:00:43.906979    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-9395f82e-serving-cert: secret "maas-api-e2e-mt-9395f82e-serving-cert" not found
Jul 06 17:00:43.907060 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:00:43.907052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls podName:dfe06055-49aa-4680-bc0c-a53b25f21552 nodeName:}" failed. No retries permitted until 2026-07-06 17:00:44.407032321 +0000 UTC m=+1921.844543213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls") pod "maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s" (UID: "dfe06055-49aa-4680-bc0c-a53b25f21552") : secret "maas-api-e2e-mt-9395f82e-serving-cert" not found
Jul 06 17:00:43.907524 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.907485    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe06055-49aa-4680-bc0c-a53b25f21552-openshift-service-ca\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:43.915786 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:43.915760    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlfk\" (UniqueName: \"kubernetes.io/projected/dfe06055-49aa-4680-bc0c-a53b25f21552-kube-api-access-mtlfk\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:44.410811 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:44.410772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:44.413295 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:44.413274    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls\") pod \"maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") " pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:44.666992 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:44.666881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:44.802871 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:44.802779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"]
Jul 06 17:00:44.805083 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:00:44.805056    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe06055_49aa_4680_bc0c_a53b25f21552.slice/crio-89f6066078a40a1640a018469d4300b2a18dfb774dd1bc742296099254b8d58c WatchSource:0}: Error finding container 89f6066078a40a1640a018469d4300b2a18dfb774dd1bc742296099254b8d58c: Status 404 returned error can't find the container with id 89f6066078a40a1640a018469d4300b2a18dfb774dd1bc742296099254b8d58c
Jul 06 17:00:44.897087 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:44.897049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s" event={"ID":"dfe06055-49aa-4680-bc0c-a53b25f21552","Type":"ContainerStarted","Data":"89f6066078a40a1640a018469d4300b2a18dfb774dd1bc742296099254b8d58c"}
Jul 06 17:00:45.903619 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:45.903526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s" event={"ID":"dfe06055-49aa-4680-bc0c-a53b25f21552","Type":"ContainerStarted","Data":"997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb"}
Jul 06 17:00:45.903986 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:45.903629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:45.921729 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:45.921675    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s" podStartSLOduration=2.0797069 podStartE2EDuration="2.921655s" totalImagesPullingTime="841.9481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:00:45.919392896 +0000 UTC m=+1923.356903805" watchObservedRunningTime="2026-07-06 17:00:45.921655 +0000 UTC m=+1923.359165909"
Jul 06 17:00:51.913717 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:51.913687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:55.875086 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:55.875042    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"]
Jul 06 17:00:55.875549 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:55.875249    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s" podUID="dfe06055-49aa-4680-bc0c-a53b25f21552" containerName="maas-api" containerID="cri-o://997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb" gracePeriod=30
Jul 06 17:00:56.125342 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.125268    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:56.219313 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.219278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls\") pod \"dfe06055-49aa-4680-bc0c-a53b25f21552\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") "
Jul 06 17:00:56.219313 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.219319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtlfk\" (UniqueName: \"kubernetes.io/projected/dfe06055-49aa-4680-bc0c-a53b25f21552-kube-api-access-mtlfk\") pod \"dfe06055-49aa-4680-bc0c-a53b25f21552\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") "
Jul 06 17:00:56.219620 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.219340    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe06055-49aa-4680-bc0c-a53b25f21552-openshift-service-ca\") pod \"dfe06055-49aa-4680-bc0c-a53b25f21552\" (UID: \"dfe06055-49aa-4680-bc0c-a53b25f21552\") "
Jul 06 17:00:56.219832 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.219803    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe06055-49aa-4680-bc0c-a53b25f21552-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfe06055-49aa-4680-bc0c-a53b25f21552" (UID: "dfe06055-49aa-4680-bc0c-a53b25f21552"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:00:56.221745 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.221719    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe06055-49aa-4680-bc0c-a53b25f21552-kube-api-access-mtlfk" (OuterVolumeSpecName: "kube-api-access-mtlfk") pod "dfe06055-49aa-4680-bc0c-a53b25f21552" (UID: "dfe06055-49aa-4680-bc0c-a53b25f21552"). InnerVolumeSpecName "kube-api-access-mtlfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:00:56.221846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.221762    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfe06055-49aa-4680-bc0c-a53b25f21552" (UID: "dfe06055-49aa-4680-bc0c-a53b25f21552"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:00:56.320405 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.320353    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfe06055-49aa-4680-bc0c-a53b25f21552-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:00:56.320405 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.320399    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlfk\" (UniqueName: \"kubernetes.io/projected/dfe06055-49aa-4680-bc0c-a53b25f21552-kube-api-access-mtlfk\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:00:56.320405 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.320412    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe06055-49aa-4680-bc0c-a53b25f21552-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:00:56.952686 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.952645    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfe06055-49aa-4680-bc0c-a53b25f21552" containerID="997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb" exitCode=0
Jul 06 17:00:56.952686 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.952686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s" event={"ID":"dfe06055-49aa-4680-bc0c-a53b25f21552","Type":"ContainerDied","Data":"997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb"}
Jul 06 17:00:56.953194 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.952704    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"
Jul 06 17:00:56.953194 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.952760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s" event={"ID":"dfe06055-49aa-4680-bc0c-a53b25f21552","Type":"ContainerDied","Data":"89f6066078a40a1640a018469d4300b2a18dfb774dd1bc742296099254b8d58c"}
Jul 06 17:00:56.953194 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.952782    2578 scope.go:117] "RemoveContainer" containerID="997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb"
Jul 06 17:00:56.963878 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.963852    2578 scope.go:117] "RemoveContainer" containerID="997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb"
Jul 06 17:00:56.964755 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:00:56.964730    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb\": container with ID starting with 997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb not found: ID does not exist" containerID="997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb"
Jul 06 17:00:56.964855 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.964762    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb"} err="failed to get container status \"997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb\": rpc error: code = NotFound desc = could not find container \"997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb\": container with ID starting with 997873011b8ecd5d7e0928e26c30a3a53d842c29fa8efefe61edf5a10c413ffb not found: ID does not exist"
Jul 06 17:00:56.977533 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.977485    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"]
Jul 06 17:00:56.981751 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:56.981722    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-9395f82e-69bc64474f-7nm9s"]
Jul 06 17:00:57.113382 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:00:57.113336    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfe06055-49aa-4680-bc0c-a53b25f21552" path="/var/lib/kubelet/pods/dfe06055-49aa-4680-bc0c-a53b25f21552/volumes"
Jul 06 17:01:25.891296 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.891258    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"]
Jul 06 17:01:25.891933 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.891667    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe06055-49aa-4680-bc0c-a53b25f21552" containerName="maas-api"
Jul 06 17:01:25.891933 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.891681    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe06055-49aa-4680-bc0c-a53b25f21552" containerName="maas-api"
Jul 06 17:01:25.891933 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.891772    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe06055-49aa-4680-bc0c-a53b25f21552" containerName="maas-api"
Jul 06 17:01:25.894721 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.894700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:25.897407 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.897380    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-a07fe89b-serving-cert\""
Jul 06 17:01:25.905897 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.905864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"]
Jul 06 17:01:25.997123 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.997082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcq7\" (UniqueName: \"kubernetes.io/projected/2bdaa21c-5170-4473-a281-491a46d8fc93-kube-api-access-rrcq7\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:25.997313 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.997143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdaa21c-5170-4473-a281-491a46d8fc93-openshift-service-ca\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:25.997313 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:25.997168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.098072 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.098036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcq7\" (UniqueName: \"kubernetes.io/projected/2bdaa21c-5170-4473-a281-491a46d8fc93-kube-api-access-rrcq7\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.098283 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.098094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdaa21c-5170-4473-a281-491a46d8fc93-openshift-service-ca\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.098283 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.098119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.098283 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:01:26.098225    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-a07fe89b-serving-cert: secret "maas-api-e2e-mt-a07fe89b-serving-cert" not found
Jul 06 17:01:26.098454 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:01:26.098295    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls podName:2bdaa21c-5170-4473-a281-491a46d8fc93 nodeName:}" failed. No retries permitted until 2026-07-06 17:01:26.598273142 +0000 UTC m=+1964.035784031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls") pod "maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw" (UID: "2bdaa21c-5170-4473-a281-491a46d8fc93") : secret "maas-api-e2e-mt-a07fe89b-serving-cert" not found
Jul 06 17:01:26.098766 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.098746    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdaa21c-5170-4473-a281-491a46d8fc93-openshift-service-ca\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.107476 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.107445    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcq7\" (UniqueName: \"kubernetes.io/projected/2bdaa21c-5170-4473-a281-491a46d8fc93-kube-api-access-rrcq7\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.604300 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.604260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.606901 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.606879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls\") pod \"maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") " pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.806042 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.806004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:26.953694 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:26.953665    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"]
Jul 06 17:01:26.956286 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:01:26.956250    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdaa21c_5170_4473_a281_491a46d8fc93.slice/crio-1becad944bd9254f317267ec76816cebbf1174acf4c237d9135575cb1cb888b0 WatchSource:0}: Error finding container 1becad944bd9254f317267ec76816cebbf1174acf4c237d9135575cb1cb888b0: Status 404 returned error can't find the container with id 1becad944bd9254f317267ec76816cebbf1174acf4c237d9135575cb1cb888b0
Jul 06 17:01:27.086723 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:27.086686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw" event={"ID":"2bdaa21c-5170-4473-a281-491a46d8fc93","Type":"ContainerStarted","Data":"1becad944bd9254f317267ec76816cebbf1174acf4c237d9135575cb1cb888b0"}
Jul 06 17:01:28.093355 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:28.093317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw" event={"ID":"2bdaa21c-5170-4473-a281-491a46d8fc93","Type":"ContainerStarted","Data":"03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7"}
Jul 06 17:01:28.093795 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:28.093413    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:01:28.109896 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:28.109827    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw" podStartSLOduration=2.723359218 podStartE2EDuration="3.109808409s" totalImagesPullingTime="386.449191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:01:28.109173931 +0000 UTC m=+1965.546684834" watchObservedRunningTime="2026-07-06 17:01:28.109808409 +0000 UTC m=+1965.547319318"
Jul 06 17:01:34.103914 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:01:34.103881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:02:13.387181 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.387134    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"]
Jul 06 17:02:13.387825 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.387357    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw" podUID="2bdaa21c-5170-4473-a281-491a46d8fc93" containerName="maas-api" containerID="cri-o://03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7" gracePeriod=30
Jul 06 17:02:13.634767 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.634738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:02:13.728677 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.728581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls\") pod \"2bdaa21c-5170-4473-a281-491a46d8fc93\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") "
Jul 06 17:02:13.728677 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.728647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdaa21c-5170-4473-a281-491a46d8fc93-openshift-service-ca\") pod \"2bdaa21c-5170-4473-a281-491a46d8fc93\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") "
Jul 06 17:02:13.728886 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.728691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrcq7\" (UniqueName: \"kubernetes.io/projected/2bdaa21c-5170-4473-a281-491a46d8fc93-kube-api-access-rrcq7\") pod \"2bdaa21c-5170-4473-a281-491a46d8fc93\" (UID: \"2bdaa21c-5170-4473-a281-491a46d8fc93\") "
Jul 06 17:02:13.729026 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.729004    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdaa21c-5170-4473-a281-491a46d8fc93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2bdaa21c-5170-4473-a281-491a46d8fc93" (UID: "2bdaa21c-5170-4473-a281-491a46d8fc93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:02:13.731041 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.731012    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdaa21c-5170-4473-a281-491a46d8fc93-kube-api-access-rrcq7" (OuterVolumeSpecName: "kube-api-access-rrcq7") pod "2bdaa21c-5170-4473-a281-491a46d8fc93" (UID: "2bdaa21c-5170-4473-a281-491a46d8fc93"). InnerVolumeSpecName "kube-api-access-rrcq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:02:13.731113 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.731042    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2bdaa21c-5170-4473-a281-491a46d8fc93" (UID: "2bdaa21c-5170-4473-a281-491a46d8fc93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:02:13.829952 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.829909    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdaa21c-5170-4473-a281-491a46d8fc93-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:02:13.829952 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.829943    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdaa21c-5170-4473-a281-491a46d8fc93-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:02:13.829952 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:13.829954    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcq7\" (UniqueName: \"kubernetes.io/projected/2bdaa21c-5170-4473-a281-491a46d8fc93-kube-api-access-rrcq7\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:02:14.295893 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.295848    2578 generic.go:356] "Generic (PLEG): container finished" podID="2bdaa21c-5170-4473-a281-491a46d8fc93" containerID="03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7" exitCode=0
Jul 06 17:02:14.295893 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.295891    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"
Jul 06 17:02:14.296108 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.295940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw" event={"ID":"2bdaa21c-5170-4473-a281-491a46d8fc93","Type":"ContainerDied","Data":"03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7"}
Jul 06 17:02:14.296108 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.295997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw" event={"ID":"2bdaa21c-5170-4473-a281-491a46d8fc93","Type":"ContainerDied","Data":"1becad944bd9254f317267ec76816cebbf1174acf4c237d9135575cb1cb888b0"}
Jul 06 17:02:14.296108 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.296019    2578 scope.go:117] "RemoveContainer" containerID="03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7"
Jul 06 17:02:14.305364 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.305344    2578 scope.go:117] "RemoveContainer" containerID="03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7"
Jul 06 17:02:14.305718 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:02:14.305697    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7\": container with ID starting with 03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7 not found: ID does not exist" containerID="03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7"
Jul 06 17:02:14.305787 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.305729    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7"} err="failed to get container status \"03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7\": rpc error: code = NotFound desc = could not find container \"03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7\": container with ID starting with 03944e49b134fe9da600393d0eb425bfda29356d737937c20c89fb2a9a50bad7 not found: ID does not exist"
Jul 06 17:02:14.320447 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.320413    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"]
Jul 06 17:02:14.324483 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:14.324457    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-a07fe89b-75f849788d-5nnkw"]
Jul 06 17:02:15.113006 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:02:15.112969    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdaa21c-5170-4473-a281-491a46d8fc93" path="/var/lib/kubelet/pods/2bdaa21c-5170-4473-a281-491a46d8fc93/volumes"
Jul 06 17:04:24.784771 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.784733    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"]
Jul 06 17:04:24.785273 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.785215    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdaa21c-5170-4473-a281-491a46d8fc93" containerName="maas-api"
Jul 06 17:04:24.785273 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.785230    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdaa21c-5170-4473-a281-491a46d8fc93" containerName="maas-api"
Jul 06 17:04:24.785345 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.785333    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdaa21c-5170-4473-a281-491a46d8fc93" containerName="maas-api"
Jul 06 17:04:24.788519 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.788481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.790769 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.790748    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-c7ba28-serving-cert\""
Jul 06 17:04:24.799404 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.799378    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"]
Jul 06 17:04:24.882284 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.882245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzt4\" (UniqueName: \"kubernetes.io/projected/9ec6254c-feee-4f7a-b012-e59ec3871b3e-kube-api-access-4bzt4\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.882284 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.882290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec6254c-feee-4f7a-b012-e59ec3871b3e-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.882551 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.882321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec6254c-feee-4f7a-b012-e59ec3871b3e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.983262 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.983224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzt4\" (UniqueName: \"kubernetes.io/projected/9ec6254c-feee-4f7a-b012-e59ec3871b3e-kube-api-access-4bzt4\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.983262 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.983264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec6254c-feee-4f7a-b012-e59ec3871b3e-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.983465 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.983338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec6254c-feee-4f7a-b012-e59ec3871b3e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.984086 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.984005    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec6254c-feee-4f7a-b012-e59ec3871b3e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.985904 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.985879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec6254c-feee-4f7a-b012-e59ec3871b3e-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:24.991062 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:24.991035    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzt4\" (UniqueName: \"kubernetes.io/projected/9ec6254c-feee-4f7a-b012-e59ec3871b3e-kube-api-access-4bzt4\") pod \"maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") " pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:25.101662 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:25.101624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:25.239130 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:25.239102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"]
Jul 06 17:04:25.241292 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:04:25.241268    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec6254c_feee_4f7a_b012_e59ec3871b3e.slice/crio-6dd5694b12584ef52573c8920d3e41c32c6191eb79ba175b80e6f42038a9e992 WatchSource:0}: Error finding container 6dd5694b12584ef52573c8920d3e41c32c6191eb79ba175b80e6f42038a9e992: Status 404 returned error can't find the container with id 6dd5694b12584ef52573c8920d3e41c32c6191eb79ba175b80e6f42038a9e992
Jul 06 17:04:25.242703 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:25.242682    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 17:04:25.875386 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:25.875341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2" event={"ID":"9ec6254c-feee-4f7a-b012-e59ec3871b3e","Type":"ContainerStarted","Data":"5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439"}
Jul 06 17:04:25.875814 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:25.875404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2" event={"ID":"9ec6254c-feee-4f7a-b012-e59ec3871b3e","Type":"ContainerStarted","Data":"6dd5694b12584ef52573c8920d3e41c32c6191eb79ba175b80e6f42038a9e992"}
Jul 06 17:04:25.875814 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:25.875448    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:25.892675 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:25.892628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2" podStartSLOduration=1.466066208 podStartE2EDuration="1.892609742s" totalImagesPullingTime="426.543534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:04:25.890727367 +0000 UTC m=+2143.328238277" watchObservedRunningTime="2026-07-06 17:04:25.892609742 +0000 UTC m=+2143.330120651"
Jul 06 17:04:31.886667 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:31.886633    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:04:32.875980 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:32.875942    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"]
Jul 06 17:04:32.879647 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:32.879616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:32.882210 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:32.882187    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-56aa0f-serving-cert\""
Jul 06 17:04:32.891907 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:32.891873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"]
Jul 06 17:04:32.958838 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:32.958796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrr4\" (UniqueName: \"kubernetes.io/projected/7c6d460a-d372-4add-8214-12bbf49dc7f5-kube-api-access-pzrr4\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:32.959045 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:32.958892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6d460a-d372-4add-8214-12bbf49dc7f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:32.959045 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:32.958986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.060313 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.060278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6d460a-d372-4add-8214-12bbf49dc7f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.060545 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.060334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.060545 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.060423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrr4\" (UniqueName: \"kubernetes.io/projected/7c6d460a-d372-4add-8214-12bbf49dc7f5-kube-api-access-pzrr4\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.060545 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:04:33.060450    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-56aa0f-serving-cert: secret "maas-api-e2e-inf-b-56aa0f-serving-cert" not found
Jul 06 17:04:33.060727 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:04:33.060577    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls podName:7c6d460a-d372-4add-8214-12bbf49dc7f5 nodeName:}" failed. No retries permitted until 2026-07-06 17:04:33.560554973 +0000 UTC m=+2150.998065862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls") pod "maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl" (UID: "7c6d460a-d372-4add-8214-12bbf49dc7f5") : secret "maas-api-e2e-inf-b-56aa0f-serving-cert" not found
Jul 06 17:04:33.061067 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.061048    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6d460a-d372-4add-8214-12bbf49dc7f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.072009 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.071979    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrr4\" (UniqueName: \"kubernetes.io/projected/7c6d460a-d372-4add-8214-12bbf49dc7f5-kube-api-access-pzrr4\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.564209 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.564170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.566984 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.566958    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") " pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.793802 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.793753    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:33.929122 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:33.929087    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"]
Jul 06 17:04:33.932114 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:04:33.932087    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6d460a_d372_4add_8214_12bbf49dc7f5.slice/crio-013ee96a17ca55c815098e2fab544acf8a6f3e7053bbb81eb30148c09ac8872f WatchSource:0}: Error finding container 013ee96a17ca55c815098e2fab544acf8a6f3e7053bbb81eb30148c09ac8872f: Status 404 returned error can't find the container with id 013ee96a17ca55c815098e2fab544acf8a6f3e7053bbb81eb30148c09ac8872f
Jul 06 17:04:34.919850 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:34.919818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl" event={"ID":"7c6d460a-d372-4add-8214-12bbf49dc7f5","Type":"ContainerStarted","Data":"8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc"}
Jul 06 17:04:34.920052 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:34.919892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl" event={"ID":"7c6d460a-d372-4add-8214-12bbf49dc7f5","Type":"ContainerStarted","Data":"013ee96a17ca55c815098e2fab544acf8a6f3e7053bbb81eb30148c09ac8872f"}
Jul 06 17:04:34.920052 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:34.919948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:34.935910 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:34.935846    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl" podStartSLOduration=2.594940457 podStartE2EDuration="2.935826611s" totalImagesPullingTime="340.886154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:04:34.93536965 +0000 UTC m=+2152.372880577" watchObservedRunningTime="2026-07-06 17:04:34.935826611 +0000 UTC m=+2152.373337520"
Jul 06 17:04:40.929369 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:40.929341    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:04:41.805529 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.805471    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"]
Jul 06 17:04:41.808999 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.808978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.812654 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.812627    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c7ba28\"/\"openshift-service-ca.crt\""
Jul 06 17:04:41.812797 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.812627    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c7ba28\"/\"default-dockercfg-fwcd5\""
Jul 06 17:04:41.812797 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.812627    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c7ba28\"/\"kube-root-ca.crt\""
Jul 06 17:04:41.812797 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.812670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c7ba28\"/\"test-model-c7ba28-kserve-self-signed-certs\""
Jul 06 17:04:41.818648 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.818623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"]
Jul 06 17:04:41.839384 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.839342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57fhg\" (UniqueName: \"kubernetes.io/projected/7dde310d-4fdc-486f-953e-303cc424142c-kube-api-access-57fhg\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.839607 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.839391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-tmp-dir\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.839607 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.839467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dde310d-4fdc-486f-953e-303cc424142c-tls-certs\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.839607 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.839537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-dshm\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.839607 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.839570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-model-cache\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.839800 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.839650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-home\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.940876 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.940838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-tmp-dir\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.940876 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.940879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dde310d-4fdc-486f-953e-303cc424142c-tls-certs\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.941430 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.940902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-dshm\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.941430 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.941026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-model-cache\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.941430 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.941111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-home\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.941430 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.941174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57fhg\" (UniqueName: \"kubernetes.io/projected/7dde310d-4fdc-486f-953e-303cc424142c-kube-api-access-57fhg\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.941430 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.941341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-tmp-dir\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.941430 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.941381    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-model-cache\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.941703 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.941476    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-home\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.943482 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.943453    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-dshm\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.943674 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.943657    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dde310d-4fdc-486f-953e-303cc424142c-tls-certs\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:41.949426 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:41.949406    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57fhg\" (UniqueName: \"kubernetes.io/projected/7dde310d-4fdc-486f-953e-303cc424142c-kube-api-access-57fhg\") pod \"test-model-c7ba28-kserve-78cdd8c8c5-2gh7c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") " pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:42.119675 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:42.119576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:42.258352 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:42.258309    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"]
Jul 06 17:04:42.260494 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:04:42.260467    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dde310d_4fdc_486f_953e_303cc424142c.slice/crio-e2b6b2fcbf23799a0f5f04f60602ab41b7b61bd2d79aff1b6675185dc2af9888 WatchSource:0}: Error finding container e2b6b2fcbf23799a0f5f04f60602ab41b7b61bd2d79aff1b6675185dc2af9888: Status 404 returned error can't find the container with id e2b6b2fcbf23799a0f5f04f60602ab41b7b61bd2d79aff1b6675185dc2af9888
Jul 06 17:04:42.958130 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:42.958094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c" event={"ID":"7dde310d-4fdc-486f-953e-303cc424142c","Type":"ContainerStarted","Data":"9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172"}
Jul 06 17:04:42.958577 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:42.958140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c" event={"ID":"7dde310d-4fdc-486f-953e-303cc424142c","Type":"ContainerStarted","Data":"e2b6b2fcbf23799a0f5f04f60602ab41b7b61bd2d79aff1b6675185dc2af9888"}
Jul 06 17:04:42.973884 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:42.973835    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c" podStartSLOduration=1.973821109 podStartE2EDuration="1.973821109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:04:42.972454574 +0000 UTC m=+2160.409965484" watchObservedRunningTime="2026-07-06 17:04:42.973821109 +0000 UTC m=+2160.411332018"
Jul 06 17:04:52.120627 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:52.120586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:52.121058 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:52.120859    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:52.133074 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:52.133043    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:04:53.013487 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:04:53.013451    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:05:08.688485 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.688456    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"]
Jul 06 17:05:08.692230 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.692208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.694872 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.694852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-56aa0f\"/\"kube-root-ca.crt\""
Jul 06 17:05:08.694991 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.694971    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-56aa0f\"/\"openshift-service-ca.crt\""
Jul 06 17:05:08.696059 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.696038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-56aa0f\"/\"test-model-56aa0f-kserve-self-signed-certs\""
Jul 06 17:05:08.696168 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.696038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-56aa0f\"/\"default-dockercfg-xbqj2\""
Jul 06 17:05:08.701221 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.701197    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"]
Jul 06 17:05:08.785480 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.785443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-dshm\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.785682 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.785494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tmp-dir\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.785682 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.785573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-model-cache\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.785682 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.785596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tls-certs\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.785682 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.785637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5fxt\" (UniqueName: \"kubernetes.io/projected/cfaf4316-b4c6-4721-b3a0-b65525dffb42-kube-api-access-w5fxt\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.785820 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.785702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-home\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.886906 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.886868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tmp-dir\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.887089 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.886932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-model-cache\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.887089 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.886973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tls-certs\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.887089 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.886998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fxt\" (UniqueName: \"kubernetes.io/projected/cfaf4316-b4c6-4721-b3a0-b65525dffb42-kube-api-access-w5fxt\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.887089 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.887048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-home\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.887290 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.887095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-dshm\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.887411 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.887385    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-model-cache\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.887470 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.887424    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tmp-dir\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.887470 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.887462    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-home\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.889410 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.889386    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-dshm\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.889684 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.889668    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tls-certs\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:08.896959 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:08.896936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5fxt\" (UniqueName: \"kubernetes.io/projected/cfaf4316-b4c6-4721-b3a0-b65525dffb42-kube-api-access-w5fxt\") pod \"test-model-56aa0f-kserve-5c59b5d6ff-7j6cr\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") " pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:09.002102 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:09.002014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:09.146255 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:09.142627    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"]
Jul 06 17:05:09.148058 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:05:09.148020    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfaf4316_b4c6_4721_b3a0_b65525dffb42.slice/crio-457079c513554ed0831a04ef837e7123079d0466b908d2100f6d508f8952bffb WatchSource:0}: Error finding container 457079c513554ed0831a04ef837e7123079d0466b908d2100f6d508f8952bffb: Status 404 returned error can't find the container with id 457079c513554ed0831a04ef837e7123079d0466b908d2100f6d508f8952bffb
Jul 06 17:05:10.085530 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:10.085470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr" event={"ID":"cfaf4316-b4c6-4721-b3a0-b65525dffb42","Type":"ContainerStarted","Data":"32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581"}
Jul 06 17:05:10.085530 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:10.085535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr" event={"ID":"cfaf4316-b4c6-4721-b3a0-b65525dffb42","Type":"ContainerStarted","Data":"457079c513554ed0831a04ef837e7123079d0466b908d2100f6d508f8952bffb"}
Jul 06 17:05:10.103736 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:10.103681    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr" podStartSLOduration=2.10366293 podStartE2EDuration="2.10366293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:05:10.101484213 +0000 UTC m=+2187.538995129" watchObservedRunningTime="2026-07-06 17:05:10.10366293 +0000 UTC m=+2187.541173839"
Jul 06 17:05:19.002268 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:19.002227    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:19.002745 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:19.002305    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:19.015076 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:19.015048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:05:19.135599 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:05:19.135572    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:06:22.379898 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.379811    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"]
Jul 06 17:06:22.383088 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.383062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.385761 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.385736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-reuse-6ae947-serving-cert\""
Jul 06 17:06:22.393932 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.393904    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"]
Jul 06 17:06:22.455936 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.455895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ab674a-f1ba-4d39-a6a6-99b7d118c149-maas-api-tls\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.455936 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.455939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76sb\" (UniqueName: \"kubernetes.io/projected/01ab674a-f1ba-4d39-a6a6-99b7d118c149-kube-api-access-s76sb\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.456148 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.455973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ab674a-f1ba-4d39-a6a6-99b7d118c149-openshift-service-ca\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.557046 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.557006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ab674a-f1ba-4d39-a6a6-99b7d118c149-maas-api-tls\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.557046 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.557051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76sb\" (UniqueName: \"kubernetes.io/projected/01ab674a-f1ba-4d39-a6a6-99b7d118c149-kube-api-access-s76sb\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.557295 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.557089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ab674a-f1ba-4d39-a6a6-99b7d118c149-openshift-service-ca\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.557973 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.557895    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ab674a-f1ba-4d39-a6a6-99b7d118c149-openshift-service-ca\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.559718 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.559696    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ab674a-f1ba-4d39-a6a6-99b7d118c149-maas-api-tls\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.566101 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.566069    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76sb\" (UniqueName: \"kubernetes.io/projected/01ab674a-f1ba-4d39-a6a6-99b7d118c149-kube-api-access-s76sb\") pod \"maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") " pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.695908 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.695806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:22.840469 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:22.840439    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"]
Jul 06 17:06:22.842986 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:06:22.842956    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ab674a_f1ba_4d39_a6a6_99b7d118c149.slice/crio-8415e4f1609265fb13fba3affc659c41571bc5c472fcb0f8784631a3c5ca4cd8 WatchSource:0}: Error finding container 8415e4f1609265fb13fba3affc659c41571bc5c472fcb0f8784631a3c5ca4cd8: Status 404 returned error can't find the container with id 8415e4f1609265fb13fba3affc659c41571bc5c472fcb0f8784631a3c5ca4cd8
Jul 06 17:06:23.419331 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:23.419293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj" event={"ID":"01ab674a-f1ba-4d39-a6a6-99b7d118c149","Type":"ContainerStarted","Data":"b92b493a8b0d1c5bdff75fedcaf3f1f824af5c2388169591d72bc1e8d23edada"}
Jul 06 17:06:23.419838 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:23.419349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj" event={"ID":"01ab674a-f1ba-4d39-a6a6-99b7d118c149","Type":"ContainerStarted","Data":"8415e4f1609265fb13fba3affc659c41571bc5c472fcb0f8784631a3c5ca4cd8"}
Jul 06 17:06:23.419838 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:23.419397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:06:23.438673 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:23.438622    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj" podStartSLOduration=1.075340571 podStartE2EDuration="1.438604707s" totalImagesPullingTime="363.264136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:06:23.436180987 +0000 UTC m=+2260.873691913" watchObservedRunningTime="2026-07-06 17:06:23.438604707 +0000 UTC m=+2260.876115615"
Jul 06 17:06:29.433547 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:06:29.433491    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:07:04.406086 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.406052    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"]
Jul 06 17:07:04.406643 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.406287    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj" podUID="01ab674a-f1ba-4d39-a6a6-99b7d118c149" containerName="maas-api" containerID="cri-o://b92b493a8b0d1c5bdff75fedcaf3f1f824af5c2388169591d72bc1e8d23edada" gracePeriod=30
Jul 06 17:07:04.429136 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.429041    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj" podUID="01ab674a-f1ba-4d39-a6a6-99b7d118c149" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.42:8443/health\": dial tcp 10.134.0.42:8443: connect: connection refused"
Jul 06 17:07:04.609961 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.609931    2578 generic.go:356] "Generic (PLEG): container finished" podID="01ab674a-f1ba-4d39-a6a6-99b7d118c149" containerID="b92b493a8b0d1c5bdff75fedcaf3f1f824af5c2388169591d72bc1e8d23edada" exitCode=0
Jul 06 17:07:04.610140 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.610005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj" event={"ID":"01ab674a-f1ba-4d39-a6a6-99b7d118c149","Type":"ContainerDied","Data":"b92b493a8b0d1c5bdff75fedcaf3f1f824af5c2388169591d72bc1e8d23edada"}
Jul 06 17:07:04.650838 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.650815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:07:04.746258 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.746172    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ab674a-f1ba-4d39-a6a6-99b7d118c149-maas-api-tls\") pod \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") "
Jul 06 17:07:04.746432 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.746263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s76sb\" (UniqueName: \"kubernetes.io/projected/01ab674a-f1ba-4d39-a6a6-99b7d118c149-kube-api-access-s76sb\") pod \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") "
Jul 06 17:07:04.746432 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.746421    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ab674a-f1ba-4d39-a6a6-99b7d118c149-openshift-service-ca\") pod \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\" (UID: \"01ab674a-f1ba-4d39-a6a6-99b7d118c149\") "
Jul 06 17:07:04.746770 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.746747    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ab674a-f1ba-4d39-a6a6-99b7d118c149-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01ab674a-f1ba-4d39-a6a6-99b7d118c149" (UID: "01ab674a-f1ba-4d39-a6a6-99b7d118c149"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:07:04.748591 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.748563    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ab674a-f1ba-4d39-a6a6-99b7d118c149-kube-api-access-s76sb" (OuterVolumeSpecName: "kube-api-access-s76sb") pod "01ab674a-f1ba-4d39-a6a6-99b7d118c149" (UID: "01ab674a-f1ba-4d39-a6a6-99b7d118c149"). InnerVolumeSpecName "kube-api-access-s76sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:07:04.748655 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.748611    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ab674a-f1ba-4d39-a6a6-99b7d118c149-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01ab674a-f1ba-4d39-a6a6-99b7d118c149" (UID: "01ab674a-f1ba-4d39-a6a6-99b7d118c149"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:07:04.847962 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.847921    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s76sb\" (UniqueName: \"kubernetes.io/projected/01ab674a-f1ba-4d39-a6a6-99b7d118c149-kube-api-access-s76sb\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:07:04.847962 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.847951    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ab674a-f1ba-4d39-a6a6-99b7d118c149-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:07:04.847962 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:04.847962    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ab674a-f1ba-4d39-a6a6-99b7d118c149-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:07:05.616232 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:05.616199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj" event={"ID":"01ab674a-f1ba-4d39-a6a6-99b7d118c149","Type":"ContainerDied","Data":"8415e4f1609265fb13fba3affc659c41571bc5c472fcb0f8784631a3c5ca4cd8"}
Jul 06 17:07:05.616672 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:05.616244    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"
Jul 06 17:07:05.616672 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:05.616259    2578 scope.go:117] "RemoveContainer" containerID="b92b493a8b0d1c5bdff75fedcaf3f1f824af5c2388169591d72bc1e8d23edada"
Jul 06 17:07:05.633075 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:05.633038    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"]
Jul 06 17:07:05.637069 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:05.637038    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-reuse-6ae947-7954fff6fc-bzcpj"]
Jul 06 17:07:07.112330 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:07.112294    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ab674a-f1ba-4d39-a6a6-99b7d118c149" path="/var/lib/kubelet/pods/01ab674a-f1ba-4d39-a6a6-99b7d118c149/volumes"
Jul 06 17:07:53.373012 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:53.372974    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"]
Jul 06 17:07:54.293090 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:54.293055    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"]
Jul 06 17:07:55.130923 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.130892    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"]
Jul 06 17:07:55.131379 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.131144    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2" podUID="9ec6254c-feee-4f7a-b012-e59ec3871b3e" containerName="maas-api" containerID="cri-o://5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439" gracePeriod=30
Jul 06 17:07:55.373868 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.373841    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:07:55.401006 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.400926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec6254c-feee-4f7a-b012-e59ec3871b3e-maas-api-tls\") pod \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") "
Jul 06 17:07:55.401141 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.401029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bzt4\" (UniqueName: \"kubernetes.io/projected/9ec6254c-feee-4f7a-b012-e59ec3871b3e-kube-api-access-4bzt4\") pod \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") "
Jul 06 17:07:55.401141 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.401121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec6254c-feee-4f7a-b012-e59ec3871b3e-openshift-service-ca\") pod \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\" (UID: \"9ec6254c-feee-4f7a-b012-e59ec3871b3e\") "
Jul 06 17:07:55.401465 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.401437    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec6254c-feee-4f7a-b012-e59ec3871b3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ec6254c-feee-4f7a-b012-e59ec3871b3e" (UID: "9ec6254c-feee-4f7a-b012-e59ec3871b3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:07:55.403235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.403206    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec6254c-feee-4f7a-b012-e59ec3871b3e-kube-api-access-4bzt4" (OuterVolumeSpecName: "kube-api-access-4bzt4") pod "9ec6254c-feee-4f7a-b012-e59ec3871b3e" (UID: "9ec6254c-feee-4f7a-b012-e59ec3871b3e"). InnerVolumeSpecName "kube-api-access-4bzt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:07:55.403361 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.403272    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec6254c-feee-4f7a-b012-e59ec3871b3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ec6254c-feee-4f7a-b012-e59ec3871b3e" (UID: "9ec6254c-feee-4f7a-b012-e59ec3871b3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:07:55.502669 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.502628    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec6254c-feee-4f7a-b012-e59ec3871b3e-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:07:55.502669 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.502657    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec6254c-feee-4f7a-b012-e59ec3871b3e-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:07:55.502669 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.502669    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzt4\" (UniqueName: \"kubernetes.io/projected/9ec6254c-feee-4f7a-b012-e59ec3871b3e-kube-api-access-4bzt4\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:07:55.849389 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.849350    2578 generic.go:356] "Generic (PLEG): container finished" podID="9ec6254c-feee-4f7a-b012-e59ec3871b3e" containerID="5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439" exitCode=0
Jul 06 17:07:55.849572 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.849401    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"
Jul 06 17:07:55.849572 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.849438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2" event={"ID":"9ec6254c-feee-4f7a-b012-e59ec3871b3e","Type":"ContainerDied","Data":"5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439"}
Jul 06 17:07:55.849572 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.849518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2" event={"ID":"9ec6254c-feee-4f7a-b012-e59ec3871b3e","Type":"ContainerDied","Data":"6dd5694b12584ef52573c8920d3e41c32c6191eb79ba175b80e6f42038a9e992"}
Jul 06 17:07:55.849572 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.849528    2578 scope.go:117] "RemoveContainer" containerID="5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439"
Jul 06 17:07:55.858390 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.858371    2578 scope.go:117] "RemoveContainer" containerID="5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439"
Jul 06 17:07:55.858673 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:07:55.858655    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439\": container with ID starting with 5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439 not found: ID does not exist" containerID="5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439"
Jul 06 17:07:55.858738 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.858682    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439"} err="failed to get container status \"5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439\": rpc error: code = NotFound desc = could not find container \"5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439\": container with ID starting with 5c5182dfa6e998a4dd7f1ec1069a040c59b0243b4cdc6e4da5362e3461d61439 not found: ID does not exist"
Jul 06 17:07:55.870738 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.870710    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"]
Jul 06 17:07:55.874343 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:55.874319    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c7ba28-59b7498bf-hpqb2"]
Jul 06 17:07:57.112607 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:07:57.112578    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ec6254c-feee-4f7a-b012-e59ec3871b3e" path="/var/lib/kubelet/pods/9ec6254c-feee-4f7a-b012-e59ec3871b3e/volumes"
Jul 06 17:08:08.393754 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.393711    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c" podUID="7dde310d-4fdc-486f-953e-303cc424142c" containerName="main" containerID="cri-o://9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172" gracePeriod=45
Jul 06 17:08:08.647770 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.647701    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:08:08.727045 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dde310d-4fdc-486f-953e-303cc424142c-tls-certs\") pod \"7dde310d-4fdc-486f-953e-303cc424142c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") "
Jul 06 17:08:08.727045 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-tmp-dir\") pod \"7dde310d-4fdc-486f-953e-303cc424142c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") "
Jul 06 17:08:08.727289 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727115    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-dshm\") pod \"7dde310d-4fdc-486f-953e-303cc424142c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") "
Jul 06 17:08:08.727289 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727137    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-home\") pod \"7dde310d-4fdc-486f-953e-303cc424142c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") "
Jul 06 17:08:08.727289 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727192    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-model-cache\") pod \"7dde310d-4fdc-486f-953e-303cc424142c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") "
Jul 06 17:08:08.727289 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57fhg\" (UniqueName: \"kubernetes.io/projected/7dde310d-4fdc-486f-953e-303cc424142c-kube-api-access-57fhg\") pod \"7dde310d-4fdc-486f-953e-303cc424142c\" (UID: \"7dde310d-4fdc-486f-953e-303cc424142c\") "
Jul 06 17:08:08.727531 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727400    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7dde310d-4fdc-486f-953e-303cc424142c" (UID: "7dde310d-4fdc-486f-953e-303cc424142c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:08:08.727531 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727437    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-home" (OuterVolumeSpecName: "home") pod "7dde310d-4fdc-486f-953e-303cc424142c" (UID: "7dde310d-4fdc-486f-953e-303cc424142c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:08:08.727531 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727453    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7dde310d-4fdc-486f-953e-303cc424142c" (UID: "7dde310d-4fdc-486f-953e-303cc424142c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:08:08.727689 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727577    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-home\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:08.727689 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727589    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-model-cache\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:08.727689 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.727600    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-tmp-dir\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:08.729647 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.729618    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dde310d-4fdc-486f-953e-303cc424142c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7dde310d-4fdc-486f-953e-303cc424142c" (UID: "7dde310d-4fdc-486f-953e-303cc424142c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:08:08.729647 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.729636    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dde310d-4fdc-486f-953e-303cc424142c-kube-api-access-57fhg" (OuterVolumeSpecName: "kube-api-access-57fhg") pod "7dde310d-4fdc-486f-953e-303cc424142c" (UID: "7dde310d-4fdc-486f-953e-303cc424142c"). InnerVolumeSpecName "kube-api-access-57fhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:08:08.729838 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.729820    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-dshm" (OuterVolumeSpecName: "dshm") pod "7dde310d-4fdc-486f-953e-303cc424142c" (UID: "7dde310d-4fdc-486f-953e-303cc424142c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:08:08.828582 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.828546    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57fhg\" (UniqueName: \"kubernetes.io/projected/7dde310d-4fdc-486f-953e-303cc424142c-kube-api-access-57fhg\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:08.828582 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.828574    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dde310d-4fdc-486f-953e-303cc424142c-tls-certs\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:08.828582 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.828585    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dde310d-4fdc-486f-953e-303cc424142c-dshm\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:08.911034 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.910941    2578 generic.go:356] "Generic (PLEG): container finished" podID="7dde310d-4fdc-486f-953e-303cc424142c" containerID="9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172" exitCode=0
Jul 06 17:08:08.911034 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.910993    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"
Jul 06 17:08:08.911034 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.911012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c" event={"ID":"7dde310d-4fdc-486f-953e-303cc424142c","Type":"ContainerDied","Data":"9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172"}
Jul 06 17:08:08.911250 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.911053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c" event={"ID":"7dde310d-4fdc-486f-953e-303cc424142c","Type":"ContainerDied","Data":"e2b6b2fcbf23799a0f5f04f60602ab41b7b61bd2d79aff1b6675185dc2af9888"}
Jul 06 17:08:08.911250 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.911072    2578 scope.go:117] "RemoveContainer" containerID="9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172"
Jul 06 17:08:08.919605 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.919587    2578 scope.go:117] "RemoveContainer" containerID="9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172"
Jul 06 17:08:08.919852 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:08:08.919834    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172\": container with ID starting with 9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172 not found: ID does not exist" containerID="9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172"
Jul 06 17:08:08.919900 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.919863    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172"} err="failed to get container status \"9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172\": rpc error: code = NotFound desc = could not find container \"9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172\": container with ID starting with 9d6fe78681205a307c76263dc198acd22aef91f40381955dddaf87a40544b172 not found: ID does not exist"
Jul 06 17:08:08.931591 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.931562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"]
Jul 06 17:08:08.935207 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:08.935183    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c7ba28/test-model-c7ba28-kserve-78cdd8c8c5-2gh7c"]
Jul 06 17:08:09.112994 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:09.112956    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dde310d-4fdc-486f-953e-303cc424142c" path="/var/lib/kubelet/pods/7dde310d-4fdc-486f-953e-303cc424142c/volumes"
Jul 06 17:08:09.864390 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:09.864344    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr" podUID="cfaf4316-b4c6-4721-b3a0-b65525dffb42" containerName="main" containerID="cri-o://32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581" gracePeriod=45
Jul 06 17:08:10.108932 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.108906    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:08:10.243283 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243191    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tls-certs\") pod \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") "
Jul 06 17:08:10.243283 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-home\") pod \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") "
Jul 06 17:08:10.243546 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-model-cache\") pod \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") "
Jul 06 17:08:10.243546 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-dshm\") pod \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") "
Jul 06 17:08:10.243546 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243329    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tmp-dir\") pod \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") "
Jul 06 17:08:10.243546 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5fxt\" (UniqueName: \"kubernetes.io/projected/cfaf4316-b4c6-4721-b3a0-b65525dffb42-kube-api-access-w5fxt\") pod \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\" (UID: \"cfaf4316-b4c6-4721-b3a0-b65525dffb42\") "
Jul 06 17:08:10.243752 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243632    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfaf4316-b4c6-4721-b3a0-b65525dffb42" (UID: "cfaf4316-b4c6-4721-b3a0-b65525dffb42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:08:10.243752 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243647    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfaf4316-b4c6-4721-b3a0-b65525dffb42" (UID: "cfaf4316-b4c6-4721-b3a0-b65525dffb42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:08:10.243752 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.243658    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-home" (OuterVolumeSpecName: "home") pod "cfaf4316-b4c6-4721-b3a0-b65525dffb42" (UID: "cfaf4316-b4c6-4721-b3a0-b65525dffb42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:08:10.245753 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.245730    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaf4316-b4c6-4721-b3a0-b65525dffb42-kube-api-access-w5fxt" (OuterVolumeSpecName: "kube-api-access-w5fxt") pod "cfaf4316-b4c6-4721-b3a0-b65525dffb42" (UID: "cfaf4316-b4c6-4721-b3a0-b65525dffb42"). InnerVolumeSpecName "kube-api-access-w5fxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:08:10.245818 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.245770    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-dshm" (OuterVolumeSpecName: "dshm") pod "cfaf4316-b4c6-4721-b3a0-b65525dffb42" (UID: "cfaf4316-b4c6-4721-b3a0-b65525dffb42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:08:10.245818 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.245775    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfaf4316-b4c6-4721-b3a0-b65525dffb42" (UID: "cfaf4316-b4c6-4721-b3a0-b65525dffb42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:08:10.344577 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.344535    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-home\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:10.344577 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.344569    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-model-cache\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:10.344577 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.344581    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-dshm\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:10.344820 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.344591    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tmp-dir\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:10.344820 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.344606    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5fxt\" (UniqueName: \"kubernetes.io/projected/cfaf4316-b4c6-4721-b3a0-b65525dffb42-kube-api-access-w5fxt\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:10.344820 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.344616    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaf4316-b4c6-4721-b3a0-b65525dffb42-tls-certs\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:10.921820 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.921783    2578 generic.go:356] "Generic (PLEG): container finished" podID="cfaf4316-b4c6-4721-b3a0-b65525dffb42" containerID="32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581" exitCode=0
Jul 06 17:08:10.922290 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.921845    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"
Jul 06 17:08:10.922290 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.921849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr" event={"ID":"cfaf4316-b4c6-4721-b3a0-b65525dffb42","Type":"ContainerDied","Data":"32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581"}
Jul 06 17:08:10.922290 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.921954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr" event={"ID":"cfaf4316-b4c6-4721-b3a0-b65525dffb42","Type":"ContainerDied","Data":"457079c513554ed0831a04ef837e7123079d0466b908d2100f6d508f8952bffb"}
Jul 06 17:08:10.922290 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.921970    2578 scope.go:117] "RemoveContainer" containerID="32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581"
Jul 06 17:08:10.931083 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.931064    2578 scope.go:117] "RemoveContainer" containerID="32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581"
Jul 06 17:08:10.931394 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:08:10.931376    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581\": container with ID starting with 32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581 not found: ID does not exist" containerID="32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581"
Jul 06 17:08:10.931467 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.931400    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581"} err="failed to get container status \"32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581\": rpc error: code = NotFound desc = could not find container \"32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581\": container with ID starting with 32866e8e8da49d86afd100ac3473a98ea95c2355c5a5dbd276d75a3d1dd8d581 not found: ID does not exist"
Jul 06 17:08:10.942929 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.942903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"]
Jul 06 17:08:10.946237 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:10.946208    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-56aa0f/test-model-56aa0f-kserve-5c59b5d6ff-7j6cr"]
Jul 06 17:08:11.113039 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:11.113006    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfaf4316-b4c6-4721-b3a0-b65525dffb42" path="/var/lib/kubelet/pods/cfaf4316-b4c6-4721-b3a0-b65525dffb42/volumes"
Jul 06 17:08:38.025401 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.025363    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"]
Jul 06 17:08:38.048434 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.048381    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl" podUID="7c6d460a-d372-4add-8214-12bbf49dc7f5" containerName="maas-api" containerID="cri-o://8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc" gracePeriod=30
Jul 06 17:08:38.294579 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.294549    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:08:38.406758 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.406722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzrr4\" (UniqueName: \"kubernetes.io/projected/7c6d460a-d372-4add-8214-12bbf49dc7f5-kube-api-access-pzrr4\") pod \"7c6d460a-d372-4add-8214-12bbf49dc7f5\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") "
Jul 06 17:08:38.406758 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.406772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6d460a-d372-4add-8214-12bbf49dc7f5-openshift-service-ca\") pod \"7c6d460a-d372-4add-8214-12bbf49dc7f5\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") "
Jul 06 17:08:38.407048 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.406844    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls\") pod \"7c6d460a-d372-4add-8214-12bbf49dc7f5\" (UID: \"7c6d460a-d372-4add-8214-12bbf49dc7f5\") "
Jul 06 17:08:38.407227 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.407203    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6d460a-d372-4add-8214-12bbf49dc7f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c6d460a-d372-4add-8214-12bbf49dc7f5" (UID: "7c6d460a-d372-4add-8214-12bbf49dc7f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:08:38.409132 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.409097    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c6d460a-d372-4add-8214-12bbf49dc7f5" (UID: "7c6d460a-d372-4add-8214-12bbf49dc7f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:08:38.409227 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.409102    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6d460a-d372-4add-8214-12bbf49dc7f5-kube-api-access-pzrr4" (OuterVolumeSpecName: "kube-api-access-pzrr4") pod "7c6d460a-d372-4add-8214-12bbf49dc7f5" (UID: "7c6d460a-d372-4add-8214-12bbf49dc7f5"). InnerVolumeSpecName "kube-api-access-pzrr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:08:38.508315 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.508268    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzrr4\" (UniqueName: \"kubernetes.io/projected/7c6d460a-d372-4add-8214-12bbf49dc7f5-kube-api-access-pzrr4\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:38.508315 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.508305    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6d460a-d372-4add-8214-12bbf49dc7f5-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:38.508315 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:38.508319    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6d460a-d372-4add-8214-12bbf49dc7f5-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:08:39.053555 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.053485    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c6d460a-d372-4add-8214-12bbf49dc7f5" containerID="8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc" exitCode=0
Jul 06 17:08:39.053555 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.053557    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"
Jul 06 17:08:39.054049 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.053567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl" event={"ID":"7c6d460a-d372-4add-8214-12bbf49dc7f5","Type":"ContainerDied","Data":"8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc"}
Jul 06 17:08:39.054049 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.053653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl" event={"ID":"7c6d460a-d372-4add-8214-12bbf49dc7f5","Type":"ContainerDied","Data":"013ee96a17ca55c815098e2fab544acf8a6f3e7053bbb81eb30148c09ac8872f"}
Jul 06 17:08:39.054049 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.053677    2578 scope.go:117] "RemoveContainer" containerID="8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc"
Jul 06 17:08:39.063294 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.063268    2578 scope.go:117] "RemoveContainer" containerID="8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc"
Jul 06 17:08:39.063610 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:08:39.063589    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc\": container with ID starting with 8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc not found: ID does not exist" containerID="8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc"
Jul 06 17:08:39.063684 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.063626    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc"} err="failed to get container status \"8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc\": rpc error: code = NotFound desc = could not find container \"8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc\": container with ID starting with 8ee914aa826ff732cf325c1bdca47eb257e7e008419692117a5534c945076dcc not found: ID does not exist"
Jul 06 17:08:39.075042 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.074997    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"]
Jul 06 17:08:39.080379 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.080350    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-56aa0f-cd4b88f8-58lzl"]
Jul 06 17:08:39.112663 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:39.112633    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6d460a-d372-4add-8214-12bbf49dc7f5" path="/var/lib/kubelet/pods/7c6d460a-d372-4add-8214-12bbf49dc7f5/volumes"
Jul 06 17:08:47.971272 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971231    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"]
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971557    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dde310d-4fdc-486f-953e-303cc424142c" containerName="main"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971569    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dde310d-4fdc-486f-953e-303cc424142c" containerName="main"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971581    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfaf4316-b4c6-4721-b3a0-b65525dffb42" containerName="main"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971586    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfaf4316-b4c6-4721-b3a0-b65525dffb42" containerName="main"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971596    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ab674a-f1ba-4d39-a6a6-99b7d118c149" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971602    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ab674a-f1ba-4d39-a6a6-99b7d118c149" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971619    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ec6254c-feee-4f7a-b012-e59ec3871b3e" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971627    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ec6254c-feee-4f7a-b012-e59ec3871b3e" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971645    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6d460a-d372-4add-8214-12bbf49dc7f5" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971651    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6d460a-d372-4add-8214-12bbf49dc7f5" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971703    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ab674a-f1ba-4d39-a6a6-99b7d118c149" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971713    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6d460a-d372-4add-8214-12bbf49dc7f5" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971719    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ec6254c-feee-4f7a-b012-e59ec3871b3e" containerName="maas-api"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971725    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dde310d-4fdc-486f-953e-303cc424142c" containerName="main"
Jul 06 17:08:47.971846 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.971731    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfaf4316-b4c6-4721-b3a0-b65525dffb42" containerName="main"
Jul 06 17:08:47.974601 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.974580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:47.977060 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.977036    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-6081ae-serving-cert\""
Jul 06 17:08:47.984835 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:47.984805    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"]
Jul 06 17:08:48.095566 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.095537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.095728 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.095594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7pw\" (UniqueName: \"kubernetes.io/projected/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-kube-api-access-jw7pw\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.095728 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.095647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.196574 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.196537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.196739 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.196703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.196793 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.196769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7pw\" (UniqueName: \"kubernetes.io/projected/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-kube-api-access-jw7pw\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.196882 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:08:48.196857    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-6081ae-serving-cert: secret "maas-api-e2e-api-a-6081ae-serving-cert" not found
Jul 06 17:08:48.196961 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:08:48.196946    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls podName:1b8e5a68-2965-4dff-aa6c-2b8afb221a4c nodeName:}" failed. No retries permitted until 2026-07-06 17:08:48.69692698 +0000 UTC m=+2406.134437875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls") pod "maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5" (UID: "1b8e5a68-2965-4dff-aa6c-2b8afb221a4c") : secret "maas-api-e2e-api-a-6081ae-serving-cert" not found
Jul 06 17:08:48.197192 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.197170    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.205991 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.205957    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7pw\" (UniqueName: \"kubernetes.io/projected/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-kube-api-access-jw7pw\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.701130 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.701085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.703679 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.703649    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls\") pod \"maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") " pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:48.886861 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:48.886822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:49.020352 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:49.020325    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"]
Jul 06 17:08:49.022060 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:08:49.022036    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8e5a68_2965_4dff_aa6c_2b8afb221a4c.slice/crio-61fc247e1e5e661fab1fccf0839f88c2bfeb3bed74def3784ff9bfc76f9db39c WatchSource:0}: Error finding container 61fc247e1e5e661fab1fccf0839f88c2bfeb3bed74def3784ff9bfc76f9db39c: Status 404 returned error can't find the container with id 61fc247e1e5e661fab1fccf0839f88c2bfeb3bed74def3784ff9bfc76f9db39c
Jul 06 17:08:49.099649 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:49.099612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5" event={"ID":"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c","Type":"ContainerStarted","Data":"61fc247e1e5e661fab1fccf0839f88c2bfeb3bed74def3784ff9bfc76f9db39c"}
Jul 06 17:08:50.105473 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:50.105439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5" event={"ID":"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c","Type":"ContainerStarted","Data":"eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0"}
Jul 06 17:08:50.105945 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:50.105553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:50.122085 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:50.122030    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5" podStartSLOduration=2.256460445 podStartE2EDuration="3.122012246s" totalImagesPullingTime="865.551801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:08:50.120367886 +0000 UTC m=+2407.557878796" watchObservedRunningTime="2026-07-06 17:08:50.122012246 +0000 UTC m=+2407.559523155"
Jul 06 17:08:56.115305 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.115276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:08:56.199810 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.199779    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"]
Jul 06 17:08:56.203239 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.203220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.205690 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.205666    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-8883ef-serving-cert\""
Jul 06 17:08:56.211797 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.211775    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"]
Jul 06 17:08:56.369140 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.369030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.369140 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.369086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-openshift-service-ca\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.369344 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.369140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qc9\" (UniqueName: \"kubernetes.io/projected/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-kube-api-access-54qc9\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.470191 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.470136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qc9\" (UniqueName: \"kubernetes.io/projected/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-kube-api-access-54qc9\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.470354 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.470252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.470354 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.470274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-openshift-service-ca\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.470446 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:08:56.470391    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-8883ef-serving-cert: secret "maas-api-e2e-api-b-8883ef-serving-cert" not found
Jul 06 17:08:56.470485 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:08:56.470470    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls podName:b5c72d1a-8fce-4f99-ad9b-543b67c4ed69 nodeName:}" failed. No retries permitted until 2026-07-06 17:08:56.970451368 +0000 UTC m=+2414.407962255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls") pod "maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54" (UID: "b5c72d1a-8fce-4f99-ad9b-543b67c4ed69") : secret "maas-api-e2e-api-b-8883ef-serving-cert" not found
Jul 06 17:08:56.470909 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.470891    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-openshift-service-ca\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.479236 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.479205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qc9\" (UniqueName: \"kubernetes.io/projected/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-kube-api-access-54qc9\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.975738 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.975684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:56.978381 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:56.978347    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls\") pod \"maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") " pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:57.114913 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:57.114872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:57.252473 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:57.252416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"]
Jul 06 17:08:57.254727 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:08:57.254698    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c72d1a_8fce_4f99_ad9b_543b67c4ed69.slice/crio-515406c62c4a89c892999743dda4d8007cda07b7ee20683f63ad3cee8fda595c WatchSource:0}: Error finding container 515406c62c4a89c892999743dda4d8007cda07b7ee20683f63ad3cee8fda595c: Status 404 returned error can't find the container with id 515406c62c4a89c892999743dda4d8007cda07b7ee20683f63ad3cee8fda595c
Jul 06 17:08:58.142907 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:58.142874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54" event={"ID":"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69","Type":"ContainerStarted","Data":"1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04"}
Jul 06 17:08:58.143143 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:58.142927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54" event={"ID":"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69","Type":"ContainerStarted","Data":"515406c62c4a89c892999743dda4d8007cda07b7ee20683f63ad3cee8fda595c"}
Jul 06 17:08:58.143143 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:58.142998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:08:58.160400 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:08:58.160345    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54" podStartSLOduration=1.807503381 podStartE2EDuration="2.16032616s" totalImagesPullingTime="352.822779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:08:58.158448774 +0000 UTC m=+2415.595959684" watchObservedRunningTime="2026-07-06 17:08:58.16032616 +0000 UTC m=+2415.597837069"
Jul 06 17:09:04.153418 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:04.153338    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:09:09.412841 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.412806    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"]
Jul 06 17:09:09.413319 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.413053    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5" podUID="1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" containerName="maas-api" containerID="cri-o://eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0" gracePeriod=30
Jul 06 17:09:09.664367 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.664305    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:09:09.686019 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.685997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw7pw\" (UniqueName: \"kubernetes.io/projected/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-kube-api-access-jw7pw\") pod \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") "
Jul 06 17:09:09.686167 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.686029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls\") pod \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") "
Jul 06 17:09:09.686167 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.686064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-openshift-service-ca\") pod \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\" (UID: \"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c\") "
Jul 06 17:09:09.686519 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.686477    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" (UID: "1b8e5a68-2965-4dff-aa6c-2b8afb221a4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:09:09.688283 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.688252    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" (UID: "1b8e5a68-2965-4dff-aa6c-2b8afb221a4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:09:09.688371 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.688304    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-kube-api-access-jw7pw" (OuterVolumeSpecName: "kube-api-access-jw7pw") pod "1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" (UID: "1b8e5a68-2965-4dff-aa6c-2b8afb221a4c"). InnerVolumeSpecName "kube-api-access-jw7pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:09:09.787268 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.787229    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7pw\" (UniqueName: \"kubernetes.io/projected/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-kube-api-access-jw7pw\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:09:09.787268 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.787259    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:09:09.787268 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:09.787269    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:09:10.197265 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.197227    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" containerID="eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0" exitCode=0
Jul 06 17:09:10.197450 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.197279    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"
Jul 06 17:09:10.197450 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.197320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5" event={"ID":"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c","Type":"ContainerDied","Data":"eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0"}
Jul 06 17:09:10.197450 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.197370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5" event={"ID":"1b8e5a68-2965-4dff-aa6c-2b8afb221a4c","Type":"ContainerDied","Data":"61fc247e1e5e661fab1fccf0839f88c2bfeb3bed74def3784ff9bfc76f9db39c"}
Jul 06 17:09:10.197450 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.197394    2578 scope.go:117] "RemoveContainer" containerID="eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0"
Jul 06 17:09:10.206956 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.206934    2578 scope.go:117] "RemoveContainer" containerID="eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0"
Jul 06 17:09:10.207250 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:09:10.207231    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0\": container with ID starting with eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0 not found: ID does not exist" containerID="eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0"
Jul 06 17:09:10.207302 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.207259    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0"} err="failed to get container status \"eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0\": rpc error: code = NotFound desc = could not find container \"eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0\": container with ID starting with eb0161a9f4da9f698dd7fae9258e1b4d50977854acb20e3cec32762c76c754f0 not found: ID does not exist"
Jul 06 17:09:10.218108 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.218080    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"]
Jul 06 17:09:10.221649 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:10.221624    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6081ae-5b66f8594d-qncg5"]
Jul 06 17:09:11.118958 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:11.118917    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" path="/var/lib/kubelet/pods/1b8e5a68-2965-4dff-aa6c-2b8afb221a4c/volumes"
Jul 06 17:09:16.447104 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.447073    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"]
Jul 06 17:09:16.447563 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.447300    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54" podUID="b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" containerName="maas-api" containerID="cri-o://1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04" gracePeriod=30
Jul 06 17:09:16.689981 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.689957    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:09:16.745648 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.745551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-openshift-service-ca\") pod \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") "
Jul 06 17:09:16.745648 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.745622    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls\") pod \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") "
Jul 06 17:09:16.745883 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.745687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54qc9\" (UniqueName: \"kubernetes.io/projected/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-kube-api-access-54qc9\") pod \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\" (UID: \"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69\") "
Jul 06 17:09:16.747625 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.746314    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" (UID: "b5c72d1a-8fce-4f99-ad9b-543b67c4ed69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:09:16.748821 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.748771    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-kube-api-access-54qc9" (OuterVolumeSpecName: "kube-api-access-54qc9") pod "b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" (UID: "b5c72d1a-8fce-4f99-ad9b-543b67c4ed69"). InnerVolumeSpecName "kube-api-access-54qc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:09:16.752547 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.750611    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" (UID: "b5c72d1a-8fce-4f99-ad9b-543b67c4ed69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:09:16.847523 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.847474    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:09:16.847664 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.847546    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54qc9\" (UniqueName: \"kubernetes.io/projected/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-kube-api-access-54qc9\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:09:16.847664 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:16.847557    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:09:17.231875 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.231840    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" containerID="1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04" exitCode=0
Jul 06 17:09:17.232074 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.231904    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"
Jul 06 17:09:17.232074 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.231925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54" event={"ID":"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69","Type":"ContainerDied","Data":"1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04"}
Jul 06 17:09:17.232074 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.231987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54" event={"ID":"b5c72d1a-8fce-4f99-ad9b-543b67c4ed69","Type":"ContainerDied","Data":"515406c62c4a89c892999743dda4d8007cda07b7ee20683f63ad3cee8fda595c"}
Jul 06 17:09:17.232074 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.232017    2578 scope.go:117] "RemoveContainer" containerID="1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04"
Jul 06 17:09:17.240764 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.240743    2578 scope.go:117] "RemoveContainer" containerID="1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04"
Jul 06 17:09:17.241086 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:09:17.241068    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04\": container with ID starting with 1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04 not found: ID does not exist" containerID="1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04"
Jul 06 17:09:17.241160 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.241100    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04"} err="failed to get container status \"1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04\": rpc error: code = NotFound desc = could not find container \"1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04\": container with ID starting with 1a12192865492bb6c6d9780c436030331132c444d0a8bb2f9f1146d5e95cdc04 not found: ID does not exist"
Jul 06 17:09:17.248352 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.248316    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"]
Jul 06 17:09:17.252706 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:17.252676    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8883ef-59bb4bf67d-47p54"]
Jul 06 17:09:19.112049 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:19.112011    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" path="/var/lib/kubelet/pods/b5c72d1a-8fce-4f99-ad9b-543b67c4ed69/volumes"
Jul 06 17:09:26.282238 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.282207    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"]
Jul 06 17:09:26.282616 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.282539    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" containerName="maas-api"
Jul 06 17:09:26.282616 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.282556    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" containerName="maas-api"
Jul 06 17:09:26.282616 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.282591    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" containerName="maas-api"
Jul 06 17:09:26.282616 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.282599    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" containerName="maas-api"
Jul 06 17:09:26.283056 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.282674    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8e5a68-2965-4dff-aa6c-2b8afb221a4c" containerName="maas-api"
Jul 06 17:09:26.283056 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.282690    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c72d1a-8fce-4f99-ad9b-543b67c4ed69" containerName="maas-api"
Jul 06 17:09:26.286961 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.286931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.289619 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.289596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-0874ce-serving-cert\""
Jul 06 17:09:26.297127 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.297102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"]
Jul 06 17:09:26.333225 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.333189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.333394 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.333261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9rd\" (UniqueName: \"kubernetes.io/projected/395738c4-51fe-446a-970c-732dbeb35c42-kube-api-access-hd9rd\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.333394 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.333288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395738c4-51fe-446a-970c-732dbeb35c42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.433834 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.433801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9rd\" (UniqueName: \"kubernetes.io/projected/395738c4-51fe-446a-970c-732dbeb35c42-kube-api-access-hd9rd\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.433834 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.433840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395738c4-51fe-446a-970c-732dbeb35c42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.434031 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.433888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.434031 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:09:26.433991    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-0874ce-serving-cert: secret "maas-api-e2e-shared-a-0874ce-serving-cert" not found
Jul 06 17:09:26.434102 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:09:26.434042    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls podName:395738c4-51fe-446a-970c-732dbeb35c42 nodeName:}" failed. No retries permitted until 2026-07-06 17:09:26.934026242 +0000 UTC m=+2444.371537129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls") pod "maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h" (UID: "395738c4-51fe-446a-970c-732dbeb35c42") : secret "maas-api-e2e-shared-a-0874ce-serving-cert" not found
Jul 06 17:09:26.434546 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.434525    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395738c4-51fe-446a-970c-732dbeb35c42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.442201 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.442178    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9rd\" (UniqueName: \"kubernetes.io/projected/395738c4-51fe-446a-970c-732dbeb35c42-kube-api-access-hd9rd\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.939342 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.939302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:26.941905 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:26.941867    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls\") pod \"maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") " pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:27.197816 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:27.197724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:27.336661 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:27.336629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"]
Jul 06 17:09:27.340378 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:09:27.340229    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395738c4_51fe_446a_970c_732dbeb35c42.slice/crio-5b18f51917f9cf89e4bc77525a35171b95ea5fa6801ea35ad11638edf57c0bc2 WatchSource:0}: Error finding container 5b18f51917f9cf89e4bc77525a35171b95ea5fa6801ea35ad11638edf57c0bc2: Status 404 returned error can't find the container with id 5b18f51917f9cf89e4bc77525a35171b95ea5fa6801ea35ad11638edf57c0bc2
Jul 06 17:09:27.341946 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:27.341926    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 17:09:28.285087 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:28.285052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h" event={"ID":"395738c4-51fe-446a-970c-732dbeb35c42","Type":"ContainerStarted","Data":"7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53"}
Jul 06 17:09:28.285263 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:28.285129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h" event={"ID":"395738c4-51fe-446a-970c-732dbeb35c42","Type":"ContainerStarted","Data":"5b18f51917f9cf89e4bc77525a35171b95ea5fa6801ea35ad11638edf57c0bc2"}
Jul 06 17:09:28.285263 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:28.285167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:28.304061 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:28.304002    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h" podStartSLOduration=1.89031937 podStartE2EDuration="2.303981179s" totalImagesPullingTime="413.661809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:28.302645474 +0000 UTC m=+2445.740156398" watchObservedRunningTime="2026-07-06 17:09:28.303981179 +0000 UTC m=+2445.741492089"
Jul 06 17:09:34.291158 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.291118    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"]
Jul 06 17:09:34.295111 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.295068    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:34.296023 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.295999    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:09:34.298120 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.298101    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-d073f6-serving-cert\""
Jul 06 17:09:34.305230 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.305206    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"]
Jul 06 17:09:34.407205 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.407164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:34.407205 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.407208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wnvw\" (UniqueName: \"kubernetes.io/projected/9d70d997-ff10-4bde-a9e6-662125f300f0-kube-api-access-6wnvw\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:34.407436 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.407234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d70d997-ff10-4bde-a9e6-662125f300f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:34.508521 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.508455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:34.508671 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.508529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wnvw\" (UniqueName: \"kubernetes.io/projected/9d70d997-ff10-4bde-a9e6-662125f300f0-kube-api-access-6wnvw\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:34.508671 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.508567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d70d997-ff10-4bde-a9e6-662125f300f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:34.508671 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:09:34.508631    2578 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-d073f6-serving-cert: secret "maas-api-e2e-shared-b-d073f6-serving-cert" not found
Jul 06 17:09:34.508825 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:09:34.508727    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls podName:9d70d997-ff10-4bde-a9e6-662125f300f0 nodeName:}" failed. No retries permitted until 2026-07-06 17:09:35.008704885 +0000 UTC m=+2452.446215772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls") pod "maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l" (UID: "9d70d997-ff10-4bde-a9e6-662125f300f0") : secret "maas-api-e2e-shared-b-d073f6-serving-cert" not found
Jul 06 17:09:34.509256 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.509238    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d70d997-ff10-4bde-a9e6-662125f300f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:34.522268 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:34.522240    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wnvw\" (UniqueName: \"kubernetes.io/projected/9d70d997-ff10-4bde-a9e6-662125f300f0-kube-api-access-6wnvw\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:35.014027 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:35.013987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:35.016576 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:35.016557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") " pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:35.208287 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:35.208251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:35.343600 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:35.343572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"]
Jul 06 17:09:35.345230 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:09:35.345202    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d70d997_ff10_4bde_a9e6_662125f300f0.slice/crio-f4119499a75e4aa4d9fce2bd852c811fdb4c42574a56c996ebdfc7ad38fc8b6f WatchSource:0}: Error finding container f4119499a75e4aa4d9fce2bd852c811fdb4c42574a56c996ebdfc7ad38fc8b6f: Status 404 returned error can't find the container with id f4119499a75e4aa4d9fce2bd852c811fdb4c42574a56c996ebdfc7ad38fc8b6f
Jul 06 17:09:36.323297 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:36.323265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l" event={"ID":"9d70d997-ff10-4bde-a9e6-662125f300f0","Type":"ContainerStarted","Data":"2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04"}
Jul 06 17:09:36.323486 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:36.323417    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:36.323486 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:36.323437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l" event={"ID":"9d70d997-ff10-4bde-a9e6-662125f300f0","Type":"ContainerStarted","Data":"f4119499a75e4aa4d9fce2bd852c811fdb4c42574a56c996ebdfc7ad38fc8b6f"}
Jul 06 17:09:36.340213 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:36.340136    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l" podStartSLOduration=1.9516038519999999 podStartE2EDuration="2.340118127s" totalImagesPullingTime="388.514275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:36.339021397 +0000 UTC m=+2453.776532307" watchObservedRunningTime="2026-07-06 17:09:36.340118127 +0000 UTC m=+2453.777629037"
Jul 06 17:09:42.334453 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:42.334425    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:09:54.383310 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.383262    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"]
Jul 06 17:09:54.386864 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.386839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.390749 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.390720    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0874ce\"/\"default-dockercfg-7lm4w\""
Jul 06 17:09:54.390749 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.390741    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0874ce\"/\"kube-root-ca.crt\""
Jul 06 17:09:54.390960 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.390725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0874ce\"/\"auth-test-model-0874ce-kserve-self-signed-certs\""
Jul 06 17:09:54.390960 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.390724    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0874ce\"/\"openshift-service-ca.crt\""
Jul 06 17:09:54.395841 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.395818    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"]
Jul 06 17:09:54.489776 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.489742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-tmp-dir\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.489948 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.489783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-dshm\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.489948 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.489860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482v9\" (UniqueName: \"kubernetes.io/projected/3e4f4cf2-8125-4154-a6df-bb64ba077350-kube-api-access-482v9\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.489948 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.489900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e4f4cf2-8125-4154-a6df-bb64ba077350-tls-certs\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.489948 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.489921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-home\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.489948 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.489944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-model-cache\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591141 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-model-cache\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591331 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-tmp-dir\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591331 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-dshm\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591331 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482v9\" (UniqueName: \"kubernetes.io/projected/3e4f4cf2-8125-4154-a6df-bb64ba077350-kube-api-access-482v9\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591331 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e4f4cf2-8125-4154-a6df-bb64ba077350-tls-certs\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591579 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-home\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591643 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591590    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-tmp-dir\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591703 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591670    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-model-cache\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.591767 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.591743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-home\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.593684 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.593656    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-dshm\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.593998 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.593982    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e4f4cf2-8125-4154-a6df-bb64ba077350-tls-certs\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.600664 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.600641    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482v9\" (UniqueName: \"kubernetes.io/projected/3e4f4cf2-8125-4154-a6df-bb64ba077350-kube-api-access-482v9\") pod \"auth-test-model-0874ce-kserve-549f95d56c-t6zxk\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") " pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.697357 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.697254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:09:54.837016 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:54.836991    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"]
Jul 06 17:09:54.839184 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:09:54.839140    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e4f4cf2_8125_4154_a6df_bb64ba077350.slice/crio-a5721620261a1c8ae930e9c571337110b247b901af6e2cd362d1dc9fe70d7599 WatchSource:0}: Error finding container a5721620261a1c8ae930e9c571337110b247b901af6e2cd362d1dc9fe70d7599: Status 404 returned error can't find the container with id a5721620261a1c8ae930e9c571337110b247b901af6e2cd362d1dc9fe70d7599
Jul 06 17:09:55.410864 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:55.410829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk" event={"ID":"3e4f4cf2-8125-4154-a6df-bb64ba077350","Type":"ContainerStarted","Data":"2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf"}
Jul 06 17:09:55.410864 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:55.410873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk" event={"ID":"3e4f4cf2-8125-4154-a6df-bb64ba077350","Type":"ContainerStarted","Data":"a5721620261a1c8ae930e9c571337110b247b901af6e2cd362d1dc9fe70d7599"}
Jul 06 17:09:55.439267 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:09:55.439218    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk" podStartSLOduration=1.439201675 podStartE2EDuration="1.439201675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:55.437573555 +0000 UTC m=+2472.875084467" watchObservedRunningTime="2026-07-06 17:09:55.439201675 +0000 UTC m=+2472.876712583"
Jul 06 17:10:04.697958 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:04.697889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:10:04.698451 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:04.697991    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:10:04.711339 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:04.711304    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:10:05.469489 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:05.469461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:10:24.970350 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:24.970309    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"]
Jul 06 17:10:24.973784 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:24.973762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:24.976475 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:24.976444    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"kube-root-ca.crt\""
Jul 06 17:10:24.977691 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:24.977666    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"openshift-service-ca.crt\""
Jul 06 17:10:24.977826 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:24.977697    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"default-dockercfg-mqgst\""
Jul 06 17:10:24.977826 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:24.977811    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"auth-test-model-d073f6-kserve-self-signed-certs\""
Jul 06 17:10:24.984679 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:24.984652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"]
Jul 06 17:10:25.063124 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.063076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-home\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.063124 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.063126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-dshm\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.063392 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.063229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tmp-dir\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.063392 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.063274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7f2j\" (UniqueName: \"kubernetes.io/projected/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-kube-api-access-n7f2j\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.063392 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.063308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-model-cache\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.063392 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.063327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tls-certs\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.164416 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.164380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7f2j\" (UniqueName: \"kubernetes.io/projected/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-kube-api-access-n7f2j\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.164630 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.164423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-model-cache\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.164630 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.164554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tls-certs\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.164630 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.164598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-home\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.164630 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.164627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-dshm\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.164864 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.164691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tmp-dir\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.164917 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.164863    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-model-cache\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.165030 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.165002    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-home\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.165137 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.165045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tmp-dir\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.167063 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.167036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-dshm\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.167230 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.167214    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tls-certs\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.172106 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.172079    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7f2j\" (UniqueName: \"kubernetes.io/projected/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-kube-api-access-n7f2j\") pod \"auth-test-model-d073f6-kserve-db86c5fff-jl8tt\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") " pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.284331 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.284236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:25.423414 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.423366    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"]
Jul 06 17:10:25.425844 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:10:25.425816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe84aa5d_5da3_4f04_a3fa_a16a79139e1e.slice/crio-048e5c3d7e4d90b8d989c9fb0d50f901863b8c504e0a3d4935c31c2f926e59a9 WatchSource:0}: Error finding container 048e5c3d7e4d90b8d989c9fb0d50f901863b8c504e0a3d4935c31c2f926e59a9: Status 404 returned error can't find the container with id 048e5c3d7e4d90b8d989c9fb0d50f901863b8c504e0a3d4935c31c2f926e59a9
Jul 06 17:10:25.551640 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.551607    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt" event={"ID":"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e","Type":"ContainerStarted","Data":"fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997"}
Jul 06 17:10:25.551815 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.551650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt" event={"ID":"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e","Type":"ContainerStarted","Data":"048e5c3d7e4d90b8d989c9fb0d50f901863b8c504e0a3d4935c31c2f926e59a9"}
Jul 06 17:10:25.570581 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:25.570488    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt" podStartSLOduration=1.570470329 podStartE2EDuration="1.570470329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:10:25.56854339 +0000 UTC m=+2503.006054301" watchObservedRunningTime="2026-07-06 17:10:25.570470329 +0000 UTC m=+2503.007981282"
Jul 06 17:10:35.285184 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:35.285138    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:35.285654 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:35.285203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:35.297976 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:35.297946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:10:35.610662 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:10:35.610634    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:12:22.480171 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:22.480081    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"]
Jul 06 17:12:23.296942 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:23.296891    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"]
Jul 06 17:12:27.681840 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.681807    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"]
Jul 06 17:12:27.685139 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.685122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.687774 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.687750    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0874ce\"/\"sub-test-model-0874ce-kserve-self-signed-certs\""
Jul 06 17:12:27.697418 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.697392    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"]
Jul 06 17:12:27.800268 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.800233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-tmp-dir\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.800268 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.800273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhqq\" (UniqueName: \"kubernetes.io/projected/d2df9b57-9b6b-495a-86ac-17358b89952a-kube-api-access-qjhqq\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.800486 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.800300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-model-cache\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.800486 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.800385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-dshm\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.800486 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.800465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df9b57-9b6b-495a-86ac-17358b89952a-tls-certs\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.800605 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.800487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-home\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901320 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-model-cache\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901320 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-dshm\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901554 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df9b57-9b6b-495a-86ac-17358b89952a-tls-certs\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901554 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-home\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901554 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-tmp-dir\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901715 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhqq\" (UniqueName: \"kubernetes.io/projected/d2df9b57-9b6b-495a-86ac-17358b89952a-kube-api-access-qjhqq\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901806 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901780    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-model-cache\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901870 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901828    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-home\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.901924 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.901872    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-tmp-dir\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.903907 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.903874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-dshm\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.904050 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.904032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df9b57-9b6b-495a-86ac-17358b89952a-tls-certs\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.909904 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.909876    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhqq\" (UniqueName: \"kubernetes.io/projected/d2df9b57-9b6b-495a-86ac-17358b89952a-kube-api-access-qjhqq\") pod \"sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") " pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:27.995306 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:27.995207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:28.144632 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:28.144605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"]
Jul 06 17:12:28.146396 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:12:28.146371    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2df9b57_9b6b_495a_86ac_17358b89952a.slice/crio-b72a454b8893cacabd18310eea6b3b71d52e996feddae6730f472c397ec6dd0a WatchSource:0}: Error finding container b72a454b8893cacabd18310eea6b3b71d52e996feddae6730f472c397ec6dd0a: Status 404 returned error can't find the container with id b72a454b8893cacabd18310eea6b3b71d52e996feddae6730f472c397ec6dd0a
Jul 06 17:12:29.114936 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:29.114902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx" event={"ID":"d2df9b57-9b6b-495a-86ac-17358b89952a","Type":"ContainerStarted","Data":"3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0"}
Jul 06 17:12:29.115332 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:29.114959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx" event={"ID":"d2df9b57-9b6b-495a-86ac-17358b89952a","Type":"ContainerStarted","Data":"b72a454b8893cacabd18310eea6b3b71d52e996feddae6730f472c397ec6dd0a"}
Jul 06 17:12:29.130965 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:29.130890    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx" podStartSLOduration=2.130868403 podStartE2EDuration="2.130868403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:12:29.130685815 +0000 UTC m=+2626.568196725" watchObservedRunningTime="2026-07-06 17:12:29.130868403 +0000 UTC m=+2626.568379315"
Jul 06 17:12:37.501122 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.501073    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk" podUID="3e4f4cf2-8125-4154-a6df-bb64ba077350" containerName="main" containerID="cri-o://2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf" gracePeriod=45
Jul 06 17:12:37.755922 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.755848    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:12:37.898822 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.898784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e4f4cf2-8125-4154-a6df-bb64ba077350-tls-certs\") pod \"3e4f4cf2-8125-4154-a6df-bb64ba077350\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") "
Jul 06 17:12:37.898822 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.898831    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-tmp-dir\") pod \"3e4f4cf2-8125-4154-a6df-bb64ba077350\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") "
Jul 06 17:12:37.899100 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.898864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-model-cache\") pod \"3e4f4cf2-8125-4154-a6df-bb64ba077350\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") "
Jul 06 17:12:37.899100 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.898905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-home\") pod \"3e4f4cf2-8125-4154-a6df-bb64ba077350\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") "
Jul 06 17:12:37.899100 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.898992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-dshm\") pod \"3e4f4cf2-8125-4154-a6df-bb64ba077350\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") "
Jul 06 17:12:37.899100 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.899030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-482v9\" (UniqueName: \"kubernetes.io/projected/3e4f4cf2-8125-4154-a6df-bb64ba077350-kube-api-access-482v9\") pod \"3e4f4cf2-8125-4154-a6df-bb64ba077350\" (UID: \"3e4f4cf2-8125-4154-a6df-bb64ba077350\") "
Jul 06 17:12:37.899304 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.899143    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e4f4cf2-8125-4154-a6df-bb64ba077350" (UID: "3e4f4cf2-8125-4154-a6df-bb64ba077350"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:12:37.899304 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.899183    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e4f4cf2-8125-4154-a6df-bb64ba077350" (UID: "3e4f4cf2-8125-4154-a6df-bb64ba077350"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:12:37.899304 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.899261    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-home" (OuterVolumeSpecName: "home") pod "3e4f4cf2-8125-4154-a6df-bb64ba077350" (UID: "3e4f4cf2-8125-4154-a6df-bb64ba077350"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:12:37.899458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.899387    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-tmp-dir\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:37.899458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.899405    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-model-cache\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:37.899458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.899415    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-home\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:37.901332 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.901292    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4f4cf2-8125-4154-a6df-bb64ba077350-kube-api-access-482v9" (OuterVolumeSpecName: "kube-api-access-482v9") pod "3e4f4cf2-8125-4154-a6df-bb64ba077350" (UID: "3e4f4cf2-8125-4154-a6df-bb64ba077350"). InnerVolumeSpecName "kube-api-access-482v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:12:37.901332 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.901320    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e4f4cf2-8125-4154-a6df-bb64ba077350-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e4f4cf2-8125-4154-a6df-bb64ba077350" (UID: "3e4f4cf2-8125-4154-a6df-bb64ba077350"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:12:37.901332 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.901306    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-dshm" (OuterVolumeSpecName: "dshm") pod "3e4f4cf2-8125-4154-a6df-bb64ba077350" (UID: "3e4f4cf2-8125-4154-a6df-bb64ba077350"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:12:37.995867 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.995807    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:37.996034 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:37.995889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:38.000136 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.000107    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e4f4cf2-8125-4154-a6df-bb64ba077350-dshm\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:38.000136 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.000136    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-482v9\" (UniqueName: \"kubernetes.io/projected/3e4f4cf2-8125-4154-a6df-bb64ba077350-kube-api-access-482v9\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:38.000286 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.000151    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e4f4cf2-8125-4154-a6df-bb64ba077350-tls-certs\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:38.008998 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.008947    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:38.157792 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.157756    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e4f4cf2-8125-4154-a6df-bb64ba077350" containerID="2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf" exitCode=0
Jul 06 17:12:38.157981 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.157805    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"
Jul 06 17:12:38.157981 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.157842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk" event={"ID":"3e4f4cf2-8125-4154-a6df-bb64ba077350","Type":"ContainerDied","Data":"2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf"}
Jul 06 17:12:38.157981 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.157883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk" event={"ID":"3e4f4cf2-8125-4154-a6df-bb64ba077350","Type":"ContainerDied","Data":"a5721620261a1c8ae930e9c571337110b247b901af6e2cd362d1dc9fe70d7599"}
Jul 06 17:12:38.157981 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.157905    2578 scope.go:117] "RemoveContainer" containerID="2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf"
Jul 06 17:12:38.168408 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.168388    2578 scope.go:117] "RemoveContainer" containerID="2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf"
Jul 06 17:12:38.168778 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:12:38.168744    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf\": container with ID starting with 2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf not found: ID does not exist" containerID="2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf"
Jul 06 17:12:38.168895 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.168787    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf"} err="failed to get container status \"2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf\": rpc error: code = NotFound desc = could not find container \"2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf\": container with ID starting with 2132e7651281d8eb033546405a2cf7a6bcc5de2d164d60b992bafb3d20d696bf not found: ID does not exist"
Jul 06 17:12:38.171777 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.171752    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:12:38.178778 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.178749    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"]
Jul 06 17:12:38.182773 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.182746    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/auth-test-model-0874ce-kserve-549f95d56c-t6zxk"]
Jul 06 17:12:38.319649 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.319611    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt" podUID="fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" containerName="main" containerID="cri-o://fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997" gracePeriod=45
Jul 06 17:12:38.581124 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.581060    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:12:38.706923 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.706891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-model-cache\") pod \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") "
Jul 06 17:12:38.706923 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.706927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tls-certs\") pod \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") "
Jul 06 17:12:38.707146 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.706998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7f2j\" (UniqueName: \"kubernetes.io/projected/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-kube-api-access-n7f2j\") pod \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") "
Jul 06 17:12:38.707146 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.707054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-home\") pod \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") "
Jul 06 17:12:38.707217 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.707190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-dshm\") pod \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") "
Jul 06 17:12:38.707257 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.707233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tmp-dir\") pod \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\" (UID: \"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e\") "
Jul 06 17:12:38.707302 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.707220    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" (UID: "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:12:38.707353 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.707295    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-home" (OuterVolumeSpecName: "home") pod "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" (UID: "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:12:38.707549 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.707524    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" (UID: "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:12:38.707629 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.707534    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-home\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:38.707629 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.707576    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-model-cache\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:38.709584 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.709559    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-kube-api-access-n7f2j" (OuterVolumeSpecName: "kube-api-access-n7f2j") pod "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" (UID: "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e"). InnerVolumeSpecName "kube-api-access-n7f2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:12:38.709680 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.709590    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" (UID: "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:12:38.709680 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.709600    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-dshm" (OuterVolumeSpecName: "dshm") pod "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" (UID: "fe84aa5d-5da3-4f04-a3fa-a16a79139e1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:12:38.808361 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.808319    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tls-certs\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:38.808361 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.808354    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7f2j\" (UniqueName: \"kubernetes.io/projected/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-kube-api-access-n7f2j\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:38.808361 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.808364    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-dshm\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:38.808613 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:38.808375    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e-tmp-dir\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:12:39.113121 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.113088    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e4f4cf2-8125-4154-a6df-bb64ba077350" path="/var/lib/kubelet/pods/3e4f4cf2-8125-4154-a6df-bb64ba077350/volumes"
Jul 06 17:12:39.164656 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.164624    2578 generic.go:356] "Generic (PLEG): container finished" podID="fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" containerID="fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997" exitCode=0
Jul 06 17:12:39.164833 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.164685    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"
Jul 06 17:12:39.164833 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.164717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt" event={"ID":"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e","Type":"ContainerDied","Data":"fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997"}
Jul 06 17:12:39.164833 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.164771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt" event={"ID":"fe84aa5d-5da3-4f04-a3fa-a16a79139e1e","Type":"ContainerDied","Data":"048e5c3d7e4d90b8d989c9fb0d50f901863b8c504e0a3d4935c31c2f926e59a9"}
Jul 06 17:12:39.164833 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.164789    2578 scope.go:117] "RemoveContainer" containerID="fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997"
Jul 06 17:12:39.173792 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.173771    2578 scope.go:117] "RemoveContainer" containerID="fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997"
Jul 06 17:12:39.174110 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:12:39.174083    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997\": container with ID starting with fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997 not found: ID does not exist" containerID="fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997"
Jul 06 17:12:39.174156 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.174115    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997"} err="failed to get container status \"fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997\": rpc error: code = NotFound desc = could not find container \"fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997\": container with ID starting with fe79055fe230f1905ad65932b53264aa73f40c476fa8f27d8f5653fad137f997 not found: ID does not exist"
Jul 06 17:12:39.180138 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.180108    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"]
Jul 06 17:12:39.184598 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:39.184570    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/auth-test-model-d073f6-kserve-db86c5fff-jl8tt"]
Jul 06 17:12:41.112965 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:41.112934    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" path="/var/lib/kubelet/pods/fe84aa5d-5da3-4f04-a3fa-a16a79139e1e/volumes"
Jul 06 17:12:56.490230 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.490190    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"]
Jul 06 17:12:56.490676 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.490607    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4f4cf2-8125-4154-a6df-bb64ba077350" containerName="main"
Jul 06 17:12:56.490676 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.490622    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4f4cf2-8125-4154-a6df-bb64ba077350" containerName="main"
Jul 06 17:12:56.490676 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.490632    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" containerName="main"
Jul 06 17:12:56.490676 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.490638    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" containerName="main"
Jul 06 17:12:56.490839 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.490706    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe84aa5d-5da3-4f04-a3fa-a16a79139e1e" containerName="main"
Jul 06 17:12:56.490839 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.490716    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4f4cf2-8125-4154-a6df-bb64ba077350" containerName="main"
Jul 06 17:12:56.493627 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.493608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.496313 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.496292    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"kube-root-ca.crt\""
Jul 06 17:12:56.497570 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.497547    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"sub-test-model-d073f6-kserve-self-signed-certs\""
Jul 06 17:12:56.497685 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.497553    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"openshift-service-ca.crt\""
Jul 06 17:12:56.497685 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.497600    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"default-dockercfg-mqgst\""
Jul 06 17:12:56.503040 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.503010    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"]
Jul 06 17:12:56.672330 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.672286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmf9\" (UniqueName: \"kubernetes.io/projected/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-kube-api-access-4vmf9\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.672545 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.672372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-dshm\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.672545 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.672425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-home\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.672545 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.672464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-model-cache\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.672684 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.672573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tmp-dir\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.672684 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.672600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tls-certs\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.773641 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.773546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tmp-dir\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.773641 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.773590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tls-certs\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.773641 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.773622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmf9\" (UniqueName: \"kubernetes.io/projected/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-kube-api-access-4vmf9\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.773641 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.773648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-dshm\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.773959 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.773671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-home\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.773959 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.773705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-model-cache\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.774069 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.774025    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tmp-dir\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.774069 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.774052    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-model-cache\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.774170 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.774075    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-home\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.776102 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.776077    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-dshm\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.776397 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.776380    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tls-certs\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.781882 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.781858    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmf9\" (UniqueName: \"kubernetes.io/projected/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-kube-api-access-4vmf9\") pod \"sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") " pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.804782 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.804740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:12:56.946753 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:56.946724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"]
Jul 06 17:12:56.949296 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:12:56.949272    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8c8644e_1a32_4dac_88dd_457b5fe5e30d.slice/crio-e3db31e6bddf254555f37674b7fe8aa3e4b2b218389f824b9460699d4956708f WatchSource:0}: Error finding container e3db31e6bddf254555f37674b7fe8aa3e4b2b218389f824b9460699d4956708f: Status 404 returned error can't find the container with id e3db31e6bddf254555f37674b7fe8aa3e4b2b218389f824b9460699d4956708f
Jul 06 17:12:57.251967 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:57.251921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb" event={"ID":"a8c8644e-1a32-4dac-88dd-457b5fe5e30d","Type":"ContainerStarted","Data":"e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa"}
Jul 06 17:12:57.252154 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:57.251977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb" event={"ID":"a8c8644e-1a32-4dac-88dd-457b5fe5e30d","Type":"ContainerStarted","Data":"e3db31e6bddf254555f37674b7fe8aa3e4b2b218389f824b9460699d4956708f"}
Jul 06 17:12:57.267889 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:12:57.267819    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb" podStartSLOduration=1.267797123 podStartE2EDuration="1.267797123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:12:57.267050938 +0000 UTC m=+2654.704561855" watchObservedRunningTime="2026-07-06 17:12:57.267797123 +0000 UTC m=+2654.705308033"
Jul 06 17:13:06.805639 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:13:06.805595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:13:06.805639 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:13:06.805639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:13:06.818185 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:13:06.818158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:13:07.311822 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:13:07.311792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:13:44.814275 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:13:44.814238    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"]
Jul 06 17:13:45.693605 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:13:45.693574    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"]
Jul 06 17:13:59.835079 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:13:59.835015    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx" podUID="d2df9b57-9b6b-495a-86ac-17358b89952a" containerName="main" containerID="cri-o://3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0" gracePeriod=45
Jul 06 17:14:00.094789 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.094719    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:14:00.238910 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.238871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-tmp-dir\") pod \"d2df9b57-9b6b-495a-86ac-17358b89952a\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") "
Jul 06 17:14:00.238910 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.238915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df9b57-9b6b-495a-86ac-17358b89952a-tls-certs\") pod \"d2df9b57-9b6b-495a-86ac-17358b89952a\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") "
Jul 06 17:14:00.239139 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.238955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjhqq\" (UniqueName: \"kubernetes.io/projected/d2df9b57-9b6b-495a-86ac-17358b89952a-kube-api-access-qjhqq\") pod \"d2df9b57-9b6b-495a-86ac-17358b89952a\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") "
Jul 06 17:14:00.239139 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.238995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-dshm\") pod \"d2df9b57-9b6b-495a-86ac-17358b89952a\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") "
Jul 06 17:14:00.239139 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.239030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-home\") pod \"d2df9b57-9b6b-495a-86ac-17358b89952a\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") "
Jul 06 17:14:00.239139 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.239077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-model-cache\") pod \"d2df9b57-9b6b-495a-86ac-17358b89952a\" (UID: \"d2df9b57-9b6b-495a-86ac-17358b89952a\") "
Jul 06 17:14:00.239346 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.239220    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2df9b57-9b6b-495a-86ac-17358b89952a" (UID: "d2df9b57-9b6b-495a-86ac-17358b89952a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:14:00.239346 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.239300    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-home" (OuterVolumeSpecName: "home") pod "d2df9b57-9b6b-495a-86ac-17358b89952a" (UID: "d2df9b57-9b6b-495a-86ac-17358b89952a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:14:00.239461 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.239405    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2df9b57-9b6b-495a-86ac-17358b89952a" (UID: "d2df9b57-9b6b-495a-86ac-17358b89952a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:14:00.239461 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.239420    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-tmp-dir\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:00.239461 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.239456    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-home\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:00.241210 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.241187    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2df9b57-9b6b-495a-86ac-17358b89952a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2df9b57-9b6b-495a-86ac-17358b89952a" (UID: "d2df9b57-9b6b-495a-86ac-17358b89952a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:14:00.241706 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.241677    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-dshm" (OuterVolumeSpecName: "dshm") pod "d2df9b57-9b6b-495a-86ac-17358b89952a" (UID: "d2df9b57-9b6b-495a-86ac-17358b89952a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:14:00.241706 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.241690    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2df9b57-9b6b-495a-86ac-17358b89952a-kube-api-access-qjhqq" (OuterVolumeSpecName: "kube-api-access-qjhqq") pod "d2df9b57-9b6b-495a-86ac-17358b89952a" (UID: "d2df9b57-9b6b-495a-86ac-17358b89952a"). InnerVolumeSpecName "kube-api-access-qjhqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:14:00.340082 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.340035    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhqq\" (UniqueName: \"kubernetes.io/projected/d2df9b57-9b6b-495a-86ac-17358b89952a-kube-api-access-qjhqq\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:00.340082 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.340071    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-dshm\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:00.340082 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.340081    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df9b57-9b6b-495a-86ac-17358b89952a-model-cache\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:00.340082 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.340090    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df9b57-9b6b-495a-86ac-17358b89952a-tls-certs\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:00.539827 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.539723    2578 generic.go:356] "Generic (PLEG): container finished" podID="d2df9b57-9b6b-495a-86ac-17358b89952a" containerID="3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0" exitCode=0
Jul 06 17:14:00.539827 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.539771    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"
Jul 06 17:14:00.539827 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.539812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx" event={"ID":"d2df9b57-9b6b-495a-86ac-17358b89952a","Type":"ContainerDied","Data":"3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0"}
Jul 06 17:14:00.540110 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.539869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx" event={"ID":"d2df9b57-9b6b-495a-86ac-17358b89952a","Type":"ContainerDied","Data":"b72a454b8893cacabd18310eea6b3b71d52e996feddae6730f472c397ec6dd0a"}
Jul 06 17:14:00.540110 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.539895    2578 scope.go:117] "RemoveContainer" containerID="3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0"
Jul 06 17:14:00.556257 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.556236    2578 scope.go:117] "RemoveContainer" containerID="3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0"
Jul 06 17:14:00.556640 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:14:00.556618    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0\": container with ID starting with 3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0 not found: ID does not exist" containerID="3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0"
Jul 06 17:14:00.556726 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.556647    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0"} err="failed to get container status \"3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0\": rpc error: code = NotFound desc = could not find container \"3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0\": container with ID starting with 3499dcdfc18be9f46b251da6ed8e4f84047ef62b27a6c02554dbafd05217a9c0 not found: ID does not exist"
Jul 06 17:14:00.565146 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.565108    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"]
Jul 06 17:14:00.566979 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.566953    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0874ce/sub-test-model-0874ce-kserve-5f8f7cdbfc-cvpzx"]
Jul 06 17:14:00.714174 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.714108    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb" podUID="a8c8644e-1a32-4dac-88dd-457b5fe5e30d" containerName="main" containerID="cri-o://e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa" gracePeriod=45
Jul 06 17:14:00.978602 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:00.978573    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:14:01.045653 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.045618    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-dshm\") pod \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") "
Jul 06 17:14:01.045834 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.045666    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vmf9\" (UniqueName: \"kubernetes.io/projected/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-kube-api-access-4vmf9\") pod \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") "
Jul 06 17:14:01.045834 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.045687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-home\") pod \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") "
Jul 06 17:14:01.046066 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.046041    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-home" (OuterVolumeSpecName: "home") pod "a8c8644e-1a32-4dac-88dd-457b5fe5e30d" (UID: "a8c8644e-1a32-4dac-88dd-457b5fe5e30d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:14:01.047939 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.047913    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-dshm" (OuterVolumeSpecName: "dshm") pod "a8c8644e-1a32-4dac-88dd-457b5fe5e30d" (UID: "a8c8644e-1a32-4dac-88dd-457b5fe5e30d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:14:01.048057 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.047953    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-kube-api-access-4vmf9" (OuterVolumeSpecName: "kube-api-access-4vmf9") pod "a8c8644e-1a32-4dac-88dd-457b5fe5e30d" (UID: "a8c8644e-1a32-4dac-88dd-457b5fe5e30d"). InnerVolumeSpecName "kube-api-access-4vmf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:14:01.112527 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.112423    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2df9b57-9b6b-495a-86ac-17358b89952a" path="/var/lib/kubelet/pods/d2df9b57-9b6b-495a-86ac-17358b89952a/volumes"
Jul 06 17:14:01.146303 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146262    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-model-cache\") pod \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") "
Jul 06 17:14:01.146303 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tmp-dir\") pod \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") "
Jul 06 17:14:01.146531 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tls-certs\") pod \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\" (UID: \"a8c8644e-1a32-4dac-88dd-457b5fe5e30d\") "
Jul 06 17:14:01.146627 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146606    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8c8644e-1a32-4dac-88dd-457b5fe5e30d" (UID: "a8c8644e-1a32-4dac-88dd-457b5fe5e30d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:14:01.146671 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146618    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8c8644e-1a32-4dac-88dd-457b5fe5e30d" (UID: "a8c8644e-1a32-4dac-88dd-457b5fe5e30d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:14:01.146805 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146782    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-dshm\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:01.146805 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146805    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vmf9\" (UniqueName: \"kubernetes.io/projected/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-kube-api-access-4vmf9\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:01.146975 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146819    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-home\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:01.146975 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146831    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-model-cache\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:01.146975 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.146843    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tmp-dir\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:01.148595 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.148563    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8c8644e-1a32-4dac-88dd-457b5fe5e30d" (UID: "a8c8644e-1a32-4dac-88dd-457b5fe5e30d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:14:01.247458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.247425    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c8644e-1a32-4dac-88dd-457b5fe5e30d-tls-certs\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:14:01.546911 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.546806    2578 generic.go:356] "Generic (PLEG): container finished" podID="a8c8644e-1a32-4dac-88dd-457b5fe5e30d" containerID="e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa" exitCode=0
Jul 06 17:14:01.546911 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.546854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"
Jul 06 17:14:01.546911 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.546890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb" event={"ID":"a8c8644e-1a32-4dac-88dd-457b5fe5e30d","Type":"ContainerDied","Data":"e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa"}
Jul 06 17:14:01.547204 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.546955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb" event={"ID":"a8c8644e-1a32-4dac-88dd-457b5fe5e30d","Type":"ContainerDied","Data":"e3db31e6bddf254555f37674b7fe8aa3e4b2b218389f824b9460699d4956708f"}
Jul 06 17:14:01.547204 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.546974    2578 scope.go:117] "RemoveContainer" containerID="e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa"
Jul 06 17:14:01.555630 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.555611    2578 scope.go:117] "RemoveContainer" containerID="e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa"
Jul 06 17:14:01.555932 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:14:01.555912    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa\": container with ID starting with e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa not found: ID does not exist" containerID="e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa"
Jul 06 17:14:01.555999 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.555946    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa"} err="failed to get container status \"e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa\": rpc error: code = NotFound desc = could not find container \"e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa\": container with ID starting with e4fea75682d1eba46a224d32423d982452b14e3b554d43113404bfd0e5f801fa not found: ID does not exist"
Jul 06 17:14:01.570202 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.570171    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"]
Jul 06 17:14:01.573910 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:01.573879    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/sub-test-model-d073f6-kserve-8cd5b8c55-8b6wb"]
Jul 06 17:14:03.113166 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:03.113129    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c8644e-1a32-4dac-88dd-457b5fe5e30d" path="/var/lib/kubelet/pods/a8c8644e-1a32-4dac-88dd-457b5fe5e30d/volumes"
Jul 06 17:14:36.569792 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.569749    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"]
Jul 06 17:14:36.570346 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.570248    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2df9b57-9b6b-495a-86ac-17358b89952a" containerName="main"
Jul 06 17:14:36.570346 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.570266    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2df9b57-9b6b-495a-86ac-17358b89952a" containerName="main"
Jul 06 17:14:36.570346 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.570291    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c8644e-1a32-4dac-88dd-457b5fe5e30d" containerName="main"
Jul 06 17:14:36.570346 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.570300    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c8644e-1a32-4dac-88dd-457b5fe5e30d" containerName="main"
Jul 06 17:14:36.570588 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.570371    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2df9b57-9b6b-495a-86ac-17358b89952a" containerName="main"
Jul 06 17:14:36.570588 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.570384    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c8644e-1a32-4dac-88dd-457b5fe5e30d" containerName="main"
Jul 06 17:14:36.573435 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.573415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.577246 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.577221    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"kube-root-ca.crt\""
Jul 06 17:14:36.577246 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.577236    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"openshift-service-ca.crt\""
Jul 06 17:14:36.577423 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.577265    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"rate-test-model-d073f6-kserve-self-signed-certs\""
Jul 06 17:14:36.577423 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.577221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d073f6\"/\"default-dockercfg-mqgst\""
Jul 06 17:14:36.582932 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.582908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"]
Jul 06 17:14:36.625128 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.625096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-home\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.625312 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.625150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-model-cache\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.625312 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.625206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tls-certs\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.625312 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.625236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-dshm\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.625312 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.625255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tmp-dir\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.625312 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.625272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ntq\" (UniqueName: \"kubernetes.io/projected/d341684c-0cab-4ca1-9490-0b10c68ad7a7-kube-api-access-c7ntq\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726196 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-home\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726406 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-model-cache\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726406 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tls-certs\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726406 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-dshm\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726406 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tmp-dir\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726406 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ntq\" (UniqueName: \"kubernetes.io/projected/d341684c-0cab-4ca1-9490-0b10c68ad7a7-kube-api-access-c7ntq\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726725 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726619    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-home\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726725 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726673    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-model-cache\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.726806 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.726754    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tmp-dir\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.728759 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.728738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-dshm\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.728916 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.728899    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tls-certs\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.734214 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.734185    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ntq\" (UniqueName: \"kubernetes.io/projected/d341684c-0cab-4ca1-9490-0b10c68ad7a7-kube-api-access-c7ntq\") pod \"rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") " pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:36.883872 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:36.883773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:37.023763 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:37.023732    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"]
Jul 06 17:14:37.026432 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:14:37.026400    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd341684c_0cab_4ca1_9490_0b10c68ad7a7.slice/crio-3e4b093d2f2b2b82eb8850cde738287fb69bfe80383908b801a391b7fdaf1955 WatchSource:0}: Error finding container 3e4b093d2f2b2b82eb8850cde738287fb69bfe80383908b801a391b7fdaf1955: Status 404 returned error can't find the container with id 3e4b093d2f2b2b82eb8850cde738287fb69bfe80383908b801a391b7fdaf1955
Jul 06 17:14:37.712612 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:37.712571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl" event={"ID":"d341684c-0cab-4ca1-9490-0b10c68ad7a7","Type":"ContainerStarted","Data":"e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3"}
Jul 06 17:14:37.712612 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:37.712614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl" event={"ID":"d341684c-0cab-4ca1-9490-0b10c68ad7a7","Type":"ContainerStarted","Data":"3e4b093d2f2b2b82eb8850cde738287fb69bfe80383908b801a391b7fdaf1955"}
Jul 06 17:14:37.729081 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:37.729022    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl" podStartSLOduration=1.729006091 podStartE2EDuration="1.729006091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:14:37.726746798 +0000 UTC m=+2755.164257709" watchObservedRunningTime="2026-07-06 17:14:37.729006091 +0000 UTC m=+2755.166517020"
Jul 06 17:14:46.884806 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:46.884763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:46.884806 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:46.884818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:46.897850 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:46.897820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:14:47.767374 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:14:47.767339    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:15:21.676121 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:21.676040    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"]
Jul 06 17:15:28.095712 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.095671    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"]
Jul 06 17:15:28.096146 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.095981    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h" podUID="395738c4-51fe-446a-970c-732dbeb35c42" containerName="maas-api" containerID="cri-o://7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53" gracePeriod=30
Jul 06 17:15:28.348164 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.348091    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:15:28.392358 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.392318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls\") pod \"395738c4-51fe-446a-970c-732dbeb35c42\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") "
Jul 06 17:15:28.392571 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.392375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395738c4-51fe-446a-970c-732dbeb35c42-openshift-service-ca\") pod \"395738c4-51fe-446a-970c-732dbeb35c42\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") "
Jul 06 17:15:28.392571 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.392462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd9rd\" (UniqueName: \"kubernetes.io/projected/395738c4-51fe-446a-970c-732dbeb35c42-kube-api-access-hd9rd\") pod \"395738c4-51fe-446a-970c-732dbeb35c42\" (UID: \"395738c4-51fe-446a-970c-732dbeb35c42\") "
Jul 06 17:15:28.392910 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.392869    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395738c4-51fe-446a-970c-732dbeb35c42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "395738c4-51fe-446a-970c-732dbeb35c42" (UID: "395738c4-51fe-446a-970c-732dbeb35c42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:15:28.394840 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.394811    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "395738c4-51fe-446a-970c-732dbeb35c42" (UID: "395738c4-51fe-446a-970c-732dbeb35c42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:15:28.394840 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.394821    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395738c4-51fe-446a-970c-732dbeb35c42-kube-api-access-hd9rd" (OuterVolumeSpecName: "kube-api-access-hd9rd") pod "395738c4-51fe-446a-970c-732dbeb35c42" (UID: "395738c4-51fe-446a-970c-732dbeb35c42"). InnerVolumeSpecName "kube-api-access-hd9rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:15:28.493350 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.493302    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395738c4-51fe-446a-970c-732dbeb35c42-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:28.493350 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.493350    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395738c4-51fe-446a-970c-732dbeb35c42-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:28.493350 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.493362    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd9rd\" (UniqueName: \"kubernetes.io/projected/395738c4-51fe-446a-970c-732dbeb35c42-kube-api-access-hd9rd\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:28.944091 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.944053    2578 generic.go:356] "Generic (PLEG): container finished" podID="395738c4-51fe-446a-970c-732dbeb35c42" containerID="7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53" exitCode=0
Jul 06 17:15:28.944297 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.944098    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"
Jul 06 17:15:28.944297 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.944141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h" event={"ID":"395738c4-51fe-446a-970c-732dbeb35c42","Type":"ContainerDied","Data":"7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53"}
Jul 06 17:15:28.944297 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.944194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h" event={"ID":"395738c4-51fe-446a-970c-732dbeb35c42","Type":"ContainerDied","Data":"5b18f51917f9cf89e4bc77525a35171b95ea5fa6801ea35ad11638edf57c0bc2"}
Jul 06 17:15:28.944297 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.944216    2578 scope.go:117] "RemoveContainer" containerID="7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53"
Jul 06 17:15:28.952835 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.952813    2578 scope.go:117] "RemoveContainer" containerID="7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53"
Jul 06 17:15:28.953106 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:15:28.953087    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53\": container with ID starting with 7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53 not found: ID does not exist" containerID="7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53"
Jul 06 17:15:28.953158 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.953113    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53"} err="failed to get container status \"7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53\": rpc error: code = NotFound desc = could not find container \"7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53\": container with ID starting with 7603e20f1b3e8966db867279d82db09a86d05ef963452a4f8e05a9e2ea682b53 not found: ID does not exist"
Jul 06 17:15:28.965924 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.965889    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"]
Jul 06 17:15:28.969493 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:28.969463    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0874ce-bc4d9798f-5kd4h"]
Jul 06 17:15:29.112259 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:29.112225    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395738c4-51fe-446a-970c-732dbeb35c42" path="/var/lib/kubelet/pods/395738c4-51fe-446a-970c-732dbeb35c42/volumes"
Jul 06 17:15:36.696757 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.696714    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl" podUID="d341684c-0cab-4ca1-9490-0b10c68ad7a7" containerName="main" containerID="cri-o://e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3" gracePeriod=45
Jul 06 17:15:36.946442 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.946402    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:15:36.984397 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.984295    2578 generic.go:356] "Generic (PLEG): container finished" podID="d341684c-0cab-4ca1-9490-0b10c68ad7a7" containerID="e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3" exitCode=0
Jul 06 17:15:36.984397 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.984344    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"
Jul 06 17:15:36.984397 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.984376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl" event={"ID":"d341684c-0cab-4ca1-9490-0b10c68ad7a7","Type":"ContainerDied","Data":"e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3"}
Jul 06 17:15:36.984655 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.984424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl" event={"ID":"d341684c-0cab-4ca1-9490-0b10c68ad7a7","Type":"ContainerDied","Data":"3e4b093d2f2b2b82eb8850cde738287fb69bfe80383908b801a391b7fdaf1955"}
Jul 06 17:15:36.984655 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.984443    2578 scope.go:117] "RemoveContainer" containerID="e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3"
Jul 06 17:15:36.994201 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.994181    2578 scope.go:117] "RemoveContainer" containerID="e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3"
Jul 06 17:15:36.994466 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:15:36.994443    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3\": container with ID starting with e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3 not found: ID does not exist" containerID="e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3"
Jul 06 17:15:36.994667 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:36.994472    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3"} err="failed to get container status \"e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3\": rpc error: code = NotFound desc = could not find container \"e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3\": container with ID starting with e127fcc533c8461580b8364988b9d7067c1a8d1af9d2115bc10fcbf73a0c6bc3 not found: ID does not exist"
Jul 06 17:15:37.073863 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.073814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-home\") pod \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") "
Jul 06 17:15:37.073863 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.073862    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-model-cache\") pod \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") "
Jul 06 17:15:37.073863 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.073883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tls-certs\") pod \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") "
Jul 06 17:15:37.074175 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.073908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-dshm\") pod \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") "
Jul 06 17:15:37.074175 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.073940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tmp-dir\") pod \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") "
Jul 06 17:15:37.074175 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.073984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7ntq\" (UniqueName: \"kubernetes.io/projected/d341684c-0cab-4ca1-9490-0b10c68ad7a7-kube-api-access-c7ntq\") pod \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\" (UID: \"d341684c-0cab-4ca1-9490-0b10c68ad7a7\") "
Jul 06 17:15:37.074308 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.074168    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d341684c-0cab-4ca1-9490-0b10c68ad7a7" (UID: "d341684c-0cab-4ca1-9490-0b10c68ad7a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:15:37.074308 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.074170    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-home" (OuterVolumeSpecName: "home") pod "d341684c-0cab-4ca1-9490-0b10c68ad7a7" (UID: "d341684c-0cab-4ca1-9490-0b10c68ad7a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:15:37.074308 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.074217    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d341684c-0cab-4ca1-9490-0b10c68ad7a7" (UID: "d341684c-0cab-4ca1-9490-0b10c68ad7a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:15:37.076135 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.076105    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-dshm" (OuterVolumeSpecName: "dshm") pod "d341684c-0cab-4ca1-9490-0b10c68ad7a7" (UID: "d341684c-0cab-4ca1-9490-0b10c68ad7a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 17:15:37.076250 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.076164    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d341684c-0cab-4ca1-9490-0b10c68ad7a7" (UID: "d341684c-0cab-4ca1-9490-0b10c68ad7a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:15:37.076250 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.076196    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d341684c-0cab-4ca1-9490-0b10c68ad7a7-kube-api-access-c7ntq" (OuterVolumeSpecName: "kube-api-access-c7ntq") pod "d341684c-0cab-4ca1-9490-0b10c68ad7a7" (UID: "d341684c-0cab-4ca1-9490-0b10c68ad7a7"). InnerVolumeSpecName "kube-api-access-c7ntq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:15:37.175297 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.175261    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-home\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:37.175458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.175310    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-model-cache\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:37.175458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.175320    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tls-certs\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:37.175458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.175329    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-dshm\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:37.175458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.175337    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d341684c-0cab-4ca1-9490-0b10c68ad7a7-tmp-dir\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:37.175458 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.175345    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7ntq\" (UniqueName: \"kubernetes.io/projected/d341684c-0cab-4ca1-9490-0b10c68ad7a7-kube-api-access-c7ntq\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:37.299682 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.299645    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"]
Jul 06 17:15:37.305764 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:37.305736    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d073f6/rate-test-model-d073f6-kserve-77c84f8fd7-6qwtl"]
Jul 06 17:15:39.113053 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:39.113019    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d341684c-0cab-4ca1-9490-0b10c68ad7a7" path="/var/lib/kubelet/pods/d341684c-0cab-4ca1-9490-0b10c68ad7a7/volumes"
Jul 06 17:15:40.091408 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.091375    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"]
Jul 06 17:15:40.091687 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.091638    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l" podUID="9d70d997-ff10-4bde-a9e6-662125f300f0" containerName="maas-api" containerID="cri-o://2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04" gracePeriod=30
Jul 06 17:15:40.539471 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.539447    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:15:40.706812 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.706720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wnvw\" (UniqueName: \"kubernetes.io/projected/9d70d997-ff10-4bde-a9e6-662125f300f0-kube-api-access-6wnvw\") pod \"9d70d997-ff10-4bde-a9e6-662125f300f0\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") "
Jul 06 17:15:40.706812 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.706779    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d70d997-ff10-4bde-a9e6-662125f300f0-openshift-service-ca\") pod \"9d70d997-ff10-4bde-a9e6-662125f300f0\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") "
Jul 06 17:15:40.706812 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.706810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls\") pod \"9d70d997-ff10-4bde-a9e6-662125f300f0\" (UID: \"9d70d997-ff10-4bde-a9e6-662125f300f0\") "
Jul 06 17:15:40.707167 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.707131    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d70d997-ff10-4bde-a9e6-662125f300f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d70d997-ff10-4bde-a9e6-662125f300f0" (UID: "9d70d997-ff10-4bde-a9e6-662125f300f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 17:15:40.709031 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.708991    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d70d997-ff10-4bde-a9e6-662125f300f0" (UID: "9d70d997-ff10-4bde-a9e6-662125f300f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 17:15:40.709141 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.709119    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d70d997-ff10-4bde-a9e6-662125f300f0-kube-api-access-6wnvw" (OuterVolumeSpecName: "kube-api-access-6wnvw") pod "9d70d997-ff10-4bde-a9e6-662125f300f0" (UID: "9d70d997-ff10-4bde-a9e6-662125f300f0"). InnerVolumeSpecName "kube-api-access-6wnvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 17:15:40.808340 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.808289    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wnvw\" (UniqueName: \"kubernetes.io/projected/9d70d997-ff10-4bde-a9e6-662125f300f0-kube-api-access-6wnvw\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:40.808340 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.808333    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d70d997-ff10-4bde-a9e6-662125f300f0-openshift-service-ca\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:40.808340 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:40.808346    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d70d997-ff10-4bde-a9e6-662125f300f0-maas-api-tls\") on node \"ip-10-0-142-118.ec2.internal\" DevicePath \"\""
Jul 06 17:15:41.007767 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.007669    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d70d997-ff10-4bde-a9e6-662125f300f0" containerID="2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04" exitCode=0
Jul 06 17:15:41.007767 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.007729    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"
Jul 06 17:15:41.007767 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.007759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l" event={"ID":"9d70d997-ff10-4bde-a9e6-662125f300f0","Type":"ContainerDied","Data":"2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04"}
Jul 06 17:15:41.008049 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.007812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l" event={"ID":"9d70d997-ff10-4bde-a9e6-662125f300f0","Type":"ContainerDied","Data":"f4119499a75e4aa4d9fce2bd852c811fdb4c42574a56c996ebdfc7ad38fc8b6f"}
Jul 06 17:15:41.008049 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.007834    2578 scope.go:117] "RemoveContainer" containerID="2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04"
Jul 06 17:15:41.017487 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.017468    2578 scope.go:117] "RemoveContainer" containerID="2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04"
Jul 06 17:15:41.017787 ip-10-0-142-118 kubenswrapper[2578]: E0706 17:15:41.017769    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04\": container with ID starting with 2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04 not found: ID does not exist" containerID="2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04"
Jul 06 17:15:41.017852 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.017801    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04"} err="failed to get container status \"2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04\": rpc error: code = NotFound desc = could not find container \"2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04\": container with ID starting with 2b101557d874dccbe7a88ba6e85340796ea2db10b9707ba20f70cd0d63e91e04 not found: ID does not exist"
Jul 06 17:15:41.028892 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.028855    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"]
Jul 06 17:15:41.032494 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.032465    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d073f6-58cb65c7c5-wvd6l"]
Jul 06 17:15:41.112548 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:41.112492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d70d997-ff10-4bde-a9e6-662125f300f0" path="/var/lib/kubelet/pods/9d70d997-ff10-4bde-a9e6-662125f300f0/volumes"
Jul 06 17:15:46.643962 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:46.643924    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57c7d7f6fb-h7gqr_df8357de-5489-4946-be3c-fef1ca96f057/authorino/0.log"
Jul 06 17:15:51.861996 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:51.861958    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-755d4b45b-drtsh_531565ef-0ed0-4616-be99-075361340f05/maas-api/0.log"
Jul 06 17:15:52.687201 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:52.687167    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4svsh_78049e17-82e0-46e7-a85d-27aaf3ffcf30/manager/0.log"
Jul 06 17:15:53.095736 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:53.095698    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb6889b66-zj8vz_de1e432a-07ea-40c7-84cb-f53bdc8afd55/copy-manifests/0.log"
Jul 06 17:15:53.102512 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:53.102474    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb6889b66-zj8vz_de1e432a-07ea-40c7-84cb-f53bdc8afd55/manager/0.log"
Jul 06 17:15:54.688942 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:54.688907    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57c7d7f6fb-h7gqr_df8357de-5489-4946-be3c-fef1ca96f057/authorino/0.log"
Jul 06 17:15:56.327450 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:56.327419    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75577b88c-wzwhh_847f5594-b381-4985-8881-611abb96b885/kube-auth-proxy/0.log"
Jul 06 17:15:56.592820 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:56.592741    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kr2kb_0f14186e-48f6-47dd-885a-c49c8e74e7ec/payload-pre-processing/0.log"
Jul 06 17:15:57.238724 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:57.238692    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4qsw7_efd2d63a-11ad-4c2d-bbdc-65c5d79f6912/main/0.log"
Jul 06 17:15:57.624158 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:57.624127    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjfkzr_26db21a2-dadd-467d-b367-1c0d777bbff2/main/0.log"
Jul 06 17:15:57.881813 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:15:57.881739    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-vgqcl_eb73b8fb-55fb-45dc-a151-abe077b31001/main/0.log"
Jul 06 17:16:02.475848 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.475813    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8rtpd/must-gather-62pxv"]
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476127    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d341684c-0cab-4ca1-9490-0b10c68ad7a7" containerName="main"
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476138    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d341684c-0cab-4ca1-9490-0b10c68ad7a7" containerName="main"
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476153    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395738c4-51fe-446a-970c-732dbeb35c42" containerName="maas-api"
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476159    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="395738c4-51fe-446a-970c-732dbeb35c42" containerName="maas-api"
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476167    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d70d997-ff10-4bde-a9e6-662125f300f0" containerName="maas-api"
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476175    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d70d997-ff10-4bde-a9e6-662125f300f0" containerName="maas-api"
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476224    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d70d997-ff10-4bde-a9e6-662125f300f0" containerName="maas-api"
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476234    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d341684c-0cab-4ca1-9490-0b10c68ad7a7" containerName="main"
Jul 06 17:16:02.476235 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.476239    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="395738c4-51fe-446a-970c-732dbeb35c42" containerName="maas-api"
Jul 06 17:16:02.479413 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.479392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8rtpd/must-gather-62pxv"
Jul 06 17:16:02.482224 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.482200    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8rtpd\"/\"kube-root-ca.crt\""
Jul 06 17:16:02.483417 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.483395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8rtpd\"/\"default-dockercfg-52s7j\""
Jul 06 17:16:02.483541 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.483395    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8rtpd\"/\"openshift-service-ca.crt\""
Jul 06 17:16:02.484429 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.484411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq5jf\" (UniqueName: \"kubernetes.io/projected/a39676c5-ce2d-4cfd-a776-ee5ee00a09de-kube-api-access-zq5jf\") pod \"must-gather-62pxv\" (UID: \"a39676c5-ce2d-4cfd-a776-ee5ee00a09de\") " pod="openshift-must-gather-8rtpd/must-gather-62pxv"
Jul 06 17:16:02.484485 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.484466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a39676c5-ce2d-4cfd-a776-ee5ee00a09de-must-gather-output\") pod \"must-gather-62pxv\" (UID: \"a39676c5-ce2d-4cfd-a776-ee5ee00a09de\") " pod="openshift-must-gather-8rtpd/must-gather-62pxv"
Jul 06 17:16:02.487734 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.487710    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8rtpd/must-gather-62pxv"]
Jul 06 17:16:02.585303 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.585260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq5jf\" (UniqueName: \"kubernetes.io/projected/a39676c5-ce2d-4cfd-a776-ee5ee00a09de-kube-api-access-zq5jf\") pod \"must-gather-62pxv\" (UID: \"a39676c5-ce2d-4cfd-a776-ee5ee00a09de\") " pod="openshift-must-gather-8rtpd/must-gather-62pxv"
Jul 06 17:16:02.585487 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.585334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a39676c5-ce2d-4cfd-a776-ee5ee00a09de-must-gather-output\") pod \"must-gather-62pxv\" (UID: \"a39676c5-ce2d-4cfd-a776-ee5ee00a09de\") " pod="openshift-must-gather-8rtpd/must-gather-62pxv"
Jul 06 17:16:02.585811 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.585791    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a39676c5-ce2d-4cfd-a776-ee5ee00a09de-must-gather-output\") pod \"must-gather-62pxv\" (UID: \"a39676c5-ce2d-4cfd-a776-ee5ee00a09de\") " pod="openshift-must-gather-8rtpd/must-gather-62pxv"
Jul 06 17:16:02.593824 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.593790    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq5jf\" (UniqueName: \"kubernetes.io/projected/a39676c5-ce2d-4cfd-a776-ee5ee00a09de-kube-api-access-zq5jf\") pod \"must-gather-62pxv\" (UID: \"a39676c5-ce2d-4cfd-a776-ee5ee00a09de\") " pod="openshift-must-gather-8rtpd/must-gather-62pxv"
Jul 06 17:16:02.789352 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.789259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8rtpd/must-gather-62pxv"
Jul 06 17:16:02.931904 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.931874    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8rtpd/must-gather-62pxv"]
Jul 06 17:16:02.934418 ip-10-0-142-118 kubenswrapper[2578]: W0706 17:16:02.934381    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda39676c5_ce2d_4cfd_a776_ee5ee00a09de.slice/crio-efdc5ebdd5f569c963d42cb222ed7da1334683fe87e958ecb322f00d6a2d5ed9 WatchSource:0}: Error finding container efdc5ebdd5f569c963d42cb222ed7da1334683fe87e958ecb322f00d6a2d5ed9: Status 404 returned error can't find the container with id efdc5ebdd5f569c963d42cb222ed7da1334683fe87e958ecb322f00d6a2d5ed9
Jul 06 17:16:02.936577 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:02.936559    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 17:16:03.103793 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:03.103758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rtpd/must-gather-62pxv" event={"ID":"a39676c5-ce2d-4cfd-a776-ee5ee00a09de","Type":"ContainerStarted","Data":"efdc5ebdd5f569c963d42cb222ed7da1334683fe87e958ecb322f00d6a2d5ed9"}
Jul 06 17:16:04.113985 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:04.113891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rtpd/must-gather-62pxv" event={"ID":"a39676c5-ce2d-4cfd-a776-ee5ee00a09de","Type":"ContainerStarted","Data":"949eea4ae0ffdbce064a81fc648f23f0ec710f6ab77c70c42e0f2b55856324f9"}
Jul 06 17:16:04.113985 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:04.113949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rtpd/must-gather-62pxv" event={"ID":"a39676c5-ce2d-4cfd-a776-ee5ee00a09de","Type":"ContainerStarted","Data":"4f52a3a04c86e2f6feb6fe737861913bb1d4be9784f0841fc9257c855789b4fc"}
Jul 06 17:16:05.462326 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:05.462285    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-slpn6_36dc0470-269b-4bed-9ab1-f1c0e63b2440/global-pull-secret-syncer/0.log"
Jul 06 17:16:05.601233 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:05.601197    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ktlhf_7ea25d04-2500-4c7b-b77d-b84c3ce92344/konnectivity-agent/0.log"
Jul 06 17:16:05.714540 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:05.714453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-118.ec2.internal_9a9545f43e4880450f1a48b17d8b749c/haproxy/0.log"
Jul 06 17:16:10.588880 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:10.588835    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57c7d7f6fb-h7gqr_df8357de-5489-4946-be3c-fef1ca96f057/authorino/0.log"
Jul 06 17:16:13.027993 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:13.027945    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6l9dj_60fdec01-6df3-4101-aa1e-3063065d3c8a/kube-state-metrics/0.log"
Jul 06 17:16:13.050084 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:13.050040    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6l9dj_60fdec01-6df3-4101-aa1e-3063065d3c8a/kube-rbac-proxy-main/0.log"
Jul 06 17:16:13.077004 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:13.076965    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6l9dj_60fdec01-6df3-4101-aa1e-3063065d3c8a/kube-rbac-proxy-self/0.log"
Jul 06 17:16:13.110437 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:13.110378    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-58ccb5cf7f-h9lbh_c82c16e7-f398-4fb4-8bc8-bf94fd2a41f0/metrics-server/0.log"
Jul 06 17:16:13.166559 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:13.165723    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-btg84_45c63935-cdc6-4034-8abe-8b1d0f4ac03e/monitoring-plugin/0.log"
Jul 06 17:16:13.212107 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:13.212046    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gt97_58b98e0f-01e5-4f18-9f4c-b28986786dcb/node-exporter/0.log"
Jul 06 17:16:13.236032 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:13.236000    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gt97_58b98e0f-01e5-4f18-9f4c-b28986786dcb/kube-rbac-proxy/0.log"
Jul 06 17:16:13.261484 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:13.261450    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gt97_58b98e0f-01e5-4f18-9f4c-b28986786dcb/init-textfile/0.log"
Jul 06 17:16:15.665073 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.664993    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8rtpd/must-gather-62pxv" podStartSLOduration=12.739119549 podStartE2EDuration="13.664967718s" totalImagesPullingTime="925.848169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:16:04.130256919 +0000 UTC m=+2841.567767839" watchObservedRunningTime="2026-07-06 17:16:15.664967718 +0000 UTC m=+2853.102478628"
Jul 06 17:16:15.665903 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.665872    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"]
Jul 06 17:16:15.671692 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.671667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.679519 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.679466    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"]
Jul 06 17:16:15.730232 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.730189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-sys\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.730446 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.730266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-podres\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.730446 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.730409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-lib-modules\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.730583 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.730532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmml6\" (UniqueName: \"kubernetes.io/projected/b6e68607-7e46-454c-9156-bf7e711001e5-kube-api-access-zmml6\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.730636 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.730575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-proc\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.831805 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.831762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-proc\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.832012 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.831861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-sys\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.832012 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.831935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-podres\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.832012 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.831985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-lib-modules\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.832151 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.832067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmml6\" (UniqueName: \"kubernetes.io/projected/b6e68607-7e46-454c-9156-bf7e711001e5-kube-api-access-zmml6\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.832563 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.832495    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-proc\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.832679 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.832606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-sys\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.832738 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.832713    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-podres\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.832812 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.832795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6e68607-7e46-454c-9156-bf7e711001e5-lib-modules\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.851352 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.851313    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmml6\" (UniqueName: \"kubernetes.io/projected/b6e68607-7e46-454c-9156-bf7e711001e5-kube-api-access-zmml6\") pod \"perf-node-gather-daemonset-8cxss\" (UID: \"b6e68607-7e46-454c-9156-bf7e711001e5\") " pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:15.988160 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:15.988066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:16.190259 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:16.189798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"]
Jul 06 17:16:17.202174 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:17.201246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss" event={"ID":"b6e68607-7e46-454c-9156-bf7e711001e5","Type":"ContainerStarted","Data":"862398317a84f5ea90baae938b6f3e123f5b968639eb41cf12bb5468dbea7da0"}
Jul 06 17:16:17.202174 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:17.201316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss" event={"ID":"b6e68607-7e46-454c-9156-bf7e711001e5","Type":"ContainerStarted","Data":"23764fb2ea87dfe4740447dc844a5912300f7fea601200cde771cb20ffd60d57"}
Jul 06 17:16:17.202174 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:17.202110    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:17.217976 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:17.217907    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss" podStartSLOduration=2.217882919 podStartE2EDuration="2.217882919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:16:17.217712866 +0000 UTC m=+2854.655223776" watchObservedRunningTime="2026-07-06 17:16:17.217882919 +0000 UTC m=+2854.655393828"
Jul 06 17:16:18.267220 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:18.267190    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2nbjn_685eb0d5-e105-422a-96fd-ff597ea88343/dns/0.log"
Jul 06 17:16:18.291951 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:18.291909    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2nbjn_685eb0d5-e105-422a-96fd-ff597ea88343/kube-rbac-proxy/0.log"
Jul 06 17:16:18.413852 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:18.413816    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8k9x7_22e56b49-6747-48b3-891d-09f7547abfe4/dns-node-resolver/0.log"
Jul 06 17:16:19.011441 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:19.011413    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76dd8f5875-hqdg8_84d6e93c-2bf6-46b9-9632-8396ec425388/registry/0.log"
Jul 06 17:16:19.089609 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:19.089576    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-smb7q_f8de9897-da9a-4c99-bd4e-9bdfab9d037f/node-ca/0.log"
Jul 06 17:16:20.311400 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:20.311371    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75577b88c-wzwhh_847f5594-b381-4985-8881-611abb96b885/kube-auth-proxy/0.log"
Jul 06 17:16:20.438455 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:20.438415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kr2kb_0f14186e-48f6-47dd-885a-c49c8e74e7ec/payload-pre-processing/0.log"
Jul 06 17:16:21.263640 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:21.263574    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vgzzv_3f60b419-0fe0-4291-a2be-eea12b95a646/serve-healthcheck-canary/0.log"
Jul 06 17:16:21.781141 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:21.781113    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9wvk_51611d17-8693-4b9b-ba1d-084079679f98/kube-rbac-proxy/0.log"
Jul 06 17:16:21.805663 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:21.805630    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9wvk_51611d17-8693-4b9b-ba1d-084079679f98/exporter/0.log"
Jul 06 17:16:21.829641 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:21.829611    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9wvk_51611d17-8693-4b9b-ba1d-084079679f98/extractor/0.log"
Jul 06 17:16:24.226126 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:24.226095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8rtpd/perf-node-gather-daemonset-8cxss"
Jul 06 17:16:24.362358 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:24.362325    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-755d4b45b-drtsh_531565ef-0ed0-4616-be99-075361340f05/maas-api/0.log"
Jul 06 17:16:24.705700 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:24.705667    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4svsh_78049e17-82e0-46e7-a85d-27aaf3ffcf30/manager/0.log"
Jul 06 17:16:24.918844 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:24.918816    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb6889b66-zj8vz_de1e432a-07ea-40c7-84cb-f53bdc8afd55/manager/0.log"
Jul 06 17:16:24.943399 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:24.943356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb6889b66-zj8vz_de1e432a-07ea-40c7-84cb-f53bdc8afd55/copy-manifests/0.log"
Jul 06 17:16:26.352324 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:26.352298    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75577b88c-wzwhh_847f5594-b381-4985-8881-611abb96b885/kube-auth-proxy/0.log"
Jul 06 17:16:26.454562 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:26.454531    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kr2kb_0f14186e-48f6-47dd-885a-c49c8e74e7ec/payload-pre-processing/0.log"
Jul 06 17:16:26.512282 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:26.512248    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5db866ddc8-d9mwj_d1e237e1-58e4-45c1-87b2-66655a18d2db/manager/0.log"
Jul 06 17:16:26.541869 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:26.541840    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qnxpb_15a921f3-ff7c-4e33-8b50-7c8a409fdf9f/openshift-lws-operator/0.log"
Jul 06 17:16:28.035956 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:28.035926    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75577b88c-wzwhh_847f5594-b381-4985-8881-611abb96b885/kube-auth-proxy/0.log"
Jul 06 17:16:28.145663 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:28.145585    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kr2kb_0f14186e-48f6-47dd-885a-c49c8e74e7ec/payload-pre-processing/0.log"
Jul 06 17:16:31.650330 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:31.650297    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-zsj74_0e8b968e-950c-453a-a4e9-2c5281824706/migrator/0.log"
Jul 06 17:16:31.673378 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:31.673349    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-zsj74_0e8b968e-950c-453a-a4e9-2c5281824706/graceful-termination/0.log"
Jul 06 17:16:33.044261 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.044177    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-49bs7_39cb27f9-a0fb-49bd-9a81-0a48771a9770/kube-multus/0.log"
Jul 06 17:16:33.245398 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.245362    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7p6kj_05388e7b-2246-4681-9c88-00caef6231de/kube-multus-additional-cni-plugins/0.log"
Jul 06 17:16:33.268695 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.268671    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7p6kj_05388e7b-2246-4681-9c88-00caef6231de/egress-router-binary-copy/0.log"
Jul 06 17:16:33.293276 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.293250    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7p6kj_05388e7b-2246-4681-9c88-00caef6231de/cni-plugins/0.log"
Jul 06 17:16:33.316428 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.316397    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7p6kj_05388e7b-2246-4681-9c88-00caef6231de/bond-cni-plugin/0.log"
Jul 06 17:16:33.339630 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.339606    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7p6kj_05388e7b-2246-4681-9c88-00caef6231de/routeoverride-cni/0.log"
Jul 06 17:16:33.362006 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.361978    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7p6kj_05388e7b-2246-4681-9c88-00caef6231de/whereabouts-cni-bincopy/0.log"
Jul 06 17:16:33.384697 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.384670    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7p6kj_05388e7b-2246-4681-9c88-00caef6231de/whereabouts-cni/0.log"
Jul 06 17:16:33.676714 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.676640    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-692bf_681e3ec4-fa09-4f46-b9f1-543d1d3a816e/network-metrics-daemon/0.log"
Jul 06 17:16:33.697814 ip-10-0-142-118 kubenswrapper[2578]: I0706 17:16:33.697782    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-692bf_681e3ec4-fa09-4f46-b9f1-543d1d3a816e/kube-rbac-proxy/0.log"