Jul 16 20:33:12.776822 ip-10-0-140-200 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.173616    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182598    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182613    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182618    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182623    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182627    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182630    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182635    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:33:13.188720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182640    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182644    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182648    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182656    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182660    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182664    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182668    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182672    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182676    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182680    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182685    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182689    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182693    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182698    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182702    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182710    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182714    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182718    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182723    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182727    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:33:13.189746 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182731    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182736    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182740    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182744    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182748    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182754    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182758    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182763    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182771    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182775    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182780    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182783    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182787    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182792    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182795    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182799    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182803    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182807    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182811    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182815    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:33:13.190536 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182819    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182828    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182832    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182837    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182844    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182850    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182855    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182864    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182868    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182872    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182876    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182881    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182889    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182894    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182899    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182903    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182907    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182911    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182915    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:33:13.191090 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182919    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182923    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182927    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182931    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182935    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182940    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182948    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182952    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182955    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182959    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182963    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182967    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182972    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182977    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182982    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182987    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.182994    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183002    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183007    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183011    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:33:13.266809 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183020    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:33:13.259482 ip-10-0-140-200 systemd[1]: Started Kubernetes Kubelet.
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183025    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183028    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183034    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183040    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183044    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183048    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183052    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183057    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183065    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183069    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183073    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.183078    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190149    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190161    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190164    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190168    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190171    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190175    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:33:13.267839 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190181    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190184    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190188    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190191    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190195    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190198    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190201    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190204    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190208    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190211    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190216    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190218    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190222    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190225    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190229    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190232    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190234    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190239    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190244    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:33:13.268736 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190248    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190252    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190256    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190259    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190263    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190266    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190270    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190274    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190277    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190282    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190287    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190291    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190295    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190298    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190300    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190305    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190307    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190310    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190314    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190317    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:33:13.269608 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190321    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190324    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190328    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190330    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190333    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190335    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190340    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190343    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190347    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190349    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190354    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190357    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190362    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190366    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190369    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190371    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190378    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190382    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190388    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:33:13.270338 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190391    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190395    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190398    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190401    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190403    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190406    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190409    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190412    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190414    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190417    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190420    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190422    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190424    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190427    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190431    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190433    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190436    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190438    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190441    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190443    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:33:13.270966 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190446    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190449    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190451    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190454    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190457    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190459    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190461    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190464    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190466    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190469    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190472    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190475    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190477    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190481    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190484    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190923    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190929    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190932    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190936    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190939    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190941    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:33:13.271487 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190944    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190946    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190949    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190951    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190954    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190956    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190959    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190961    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190964    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190967    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190969    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190972    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190975    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190977    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190980    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190982    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190985    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190987    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190990    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190994    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:33:13.272091 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.190997    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191000    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191003    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191005    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191009    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191011    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191014    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191016    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191019    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191022    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191024    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191027    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191029    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191032    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191034    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191037    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191039    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191041    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191044    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191049    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:33:13.272928 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191052    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191054    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191057    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191059    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191062    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191064    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191067    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191070    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191073    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191075    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191078    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191080    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191083    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191085    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191087    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191092    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191095    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191100    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191104    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:33:13.273923 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191108    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191111    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191114    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191116    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191119    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191121    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191124    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191128    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191131    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191134    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191136    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191139    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191143    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191146    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191149    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191151    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191154    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191157    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191160    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:33:13.274439 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191162    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191165    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191167    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191171    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191174    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191176    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191178    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191181    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191184    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191187    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191189    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191192    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191195    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191197    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191200    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191271    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191278    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191285    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191290    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191296    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191301    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 20:33:13.275133 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191307    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191311    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191315    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191318    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191325    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191329    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191332    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191335    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191338    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191341    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191344    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191347    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191351    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191354    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191357    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191361    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191364    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191368    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191371    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191375    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191378    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191381    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191384    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191387    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 20:33:13.275784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191391    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191394    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191399    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191402    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191405    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191407    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191410    2580 flags.go:64] FLAG: --enable-server="true"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191413    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191417    2580 flags.go:64] FLAG: --event-burst="100"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191420    2580 flags.go:64] FLAG: --event-qps="50"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191422    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191426    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191431    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191435    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191437    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191441    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191444    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191447    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191450    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191453    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191456    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191459    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191461    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191464    2580 flags.go:64] FLAG: --feature-gates=""
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191468    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 20:33:13.276537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191471    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191474    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191477    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191480    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191483    2580 flags.go:64] FLAG: --help="false"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191486    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191489    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191491    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191495    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191498    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191501    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191503    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191506    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191524    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191529    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191534    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191537    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191540    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191543    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191549    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191552    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191555    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191558    2580 flags.go:64] FLAG: --lock-file=""
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191561    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 20:33:13.277966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191564    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191567    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191573    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191576    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191579    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191581    2580 flags.go:64] FLAG: --logging-format="text"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191584    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191588    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191591    2580 flags.go:64] FLAG: --manifest-url=""
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191594    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191598    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191601    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191605    2580 flags.go:64] FLAG: --max-pods="110"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191607    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191611    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191613    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191616    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191620    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191623    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191627    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191639    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191643    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191646    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 20:33:13.279372 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191649    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191652    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191658    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191661    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191665    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191671    2580 flags.go:64] FLAG: --port="10250"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191674    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191677    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04662b2ca6004f8a8"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191680    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191683    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191686    2580 flags.go:64] FLAG: --register-node="true"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191689    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191693    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191696    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191699    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191702    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191706    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191709    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191712    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191715    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191718    2580 flags.go:64] FLAG: --runonce="false"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191721    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191724    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191727    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191730    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 20:33:13.280391 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191732    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191736    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191740    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191743    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191745    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191748    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191751    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191754    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191757    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191759    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191762    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191767    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191771    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191774    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191778    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191781    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191784    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191787    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191790    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191793    2580 flags.go:64] FLAG: --v="2"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191797    2580 flags.go:64] FLAG: --version="false"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191801    2580 flags.go:64] FLAG: --vmodule=""
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191805    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.191808    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191927    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:33:13.281244 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191930    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191935    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191939    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191942    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191946    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191949    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191952    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191955    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191958    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191961    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191964    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191967    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191970    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191973    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191975    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191978    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191980    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191983    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191987    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:33:13.282101 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191992    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191995    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.191998    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192001    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192004    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192006    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192009    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192012    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192014    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192017    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192020    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192024    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192026    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192029    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192032    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192034    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192037    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192040    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192043    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192046    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:33:13.282742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192048    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192051    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192053    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192056    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192059    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192061    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192064    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192066    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192069    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192071    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192074    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192076    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192080    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192083    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192085    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192087    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192090    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192092    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192095    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192098    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:33:13.285720 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192100    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192103    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192106    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192109    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192112    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192114    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192117    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192119    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192122    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192125    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192128    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192130    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192133    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192135    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192138    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192140    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192143    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192146    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192148    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192151    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:33:13.286774 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192153    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192156    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192158    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192160    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192164    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192167    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192169    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192171    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192174    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192176    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192180    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192184    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192187    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192190    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192192    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192197    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192199    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192202    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:33:13.375152 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.192204    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.192940    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.204572    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.204591    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.204612    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.204618    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204691    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204696    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204699    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204702    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204705    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204709    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204712    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204715    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204718    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:33:13.376198 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204721    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204724    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204727    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204730    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204732    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204735    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204738    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204741    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204743    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204746    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204749    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204751    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204755    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204757    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204760    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204763    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204765    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204768    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204771    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204773    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:33:13.377064 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204776    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204779    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204782    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204785    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204787    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204790    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204795    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204798    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204801    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204805    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204807    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204810    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204813    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204815    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204818    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204820    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204823    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204826    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204828    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204831    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:33:13.378159 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204834    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204838    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204841    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204843    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204846    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204849    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204852    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204854    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204857    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204859    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204862    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204865    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204867    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204870    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204873    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204877    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204881    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204883    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204890    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:33:13.380742 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204893    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204896    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204899    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204902    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204905    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204908    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204911    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204914    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204917    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204920    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204922    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204925    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204927    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204930    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204934    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204937    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204940    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204942    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204945    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204947    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:33:13.381775 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204950    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204953    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204957    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204960    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204963    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204966    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204968    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204971    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204975    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204979    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.204982    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.204989    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205140    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205145    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:33:13.398359 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205148    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205151    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205155    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205158    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205160    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205163    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205165    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205168    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205171    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205173    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205177    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205181    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205184    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205188    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205190    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205193    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205196    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205199    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205201    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205204    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:33:13.399288 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205206    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205209    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205212    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205215    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205217    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205220    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205223    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205225    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205228    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205232    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205235    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205238    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205240    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205243    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205245    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205248    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205251    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205254    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205256    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205259    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:33:13.400239 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205262    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205264    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205267    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205269    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205272    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205276    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205279    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205282    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205284    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205287    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205289    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205292    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205294    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205297    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205299    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205302    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205305    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205307    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205310    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205312    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:33:13.401199 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205315    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205318    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205321    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205323    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205326    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205328    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205331    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205333    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205336    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205339    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205341    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205344    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205346    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205349    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205352    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205354    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205357    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205361    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205364    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:33:13.402143 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205367    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205371    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205374    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205377    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205380    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205383    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205385    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205388    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205391    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205393    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205396    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205398    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205401    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205404    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205407    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205410    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205414    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:33:13.403104 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:13.205417    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.205423    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.205567    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.209212    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.210556    2580 server.go:1019] "Starting client certificate rotation"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.210669    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.210698    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.213793    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.217458    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.228638    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.230692    2580 log.go:25] "Validated CRI v1 image API"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.232645    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 20:33:13.404046 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.236454    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ab208820-92f8-48b5-9de9-92b342ba8a3f:/dev/nvme0n1p4 bc73af2c-7b92-4a2d-acd8-7b7e819922c6:/dev/nvme0n1p3]
Jul 16 20:33:13.404764 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.236472    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 20:33:13.404764 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.236572    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.241025    2580 manager.go:216] Machine: {Timestamp:2026-07-16 20:33:13.240538926 +0000 UTC m=+0.343817901 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100958 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b6f1f57d89e2831ac1314e1440675 SystemUUID:ec2b6f1f-57d8-9e28-31ac-1314e1440675 BootID:912d3678-6328-48da-9c74-87b5024165a3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c7:80:9b:f1:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c7:80:9b:f1:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:48:38:84:26:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.241134    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.241212    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.242366    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.242390    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.243039    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.243048    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.243056    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.243644    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.244701    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.247393    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.247502    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.247544    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.247554    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.247568    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.247577    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.247588    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 20:33:13.405226 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.247994    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.248009    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.250915    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.252971    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.253008    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254013    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254029    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254038    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254046    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254057    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254064    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254076    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254084    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254093    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254101    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254112    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254120    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254908    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.254918    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.258767    2580 server.go:1268] "Started kubelet"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.259466    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5czq4"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.259500    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.261632    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.262069    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.262122    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.263458    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.264128    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.264153    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.264217    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.267142    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5czq4"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.270260    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.272362    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.269108    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-200.ec2.internal.18c2df5d1459f3c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-200.ec2.internal,UID:ip-10-0-140-200.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-200.ec2.internal,},FirstTimestamp:2026-07-16 20:33:13.258529735 +0000 UTC m=+0.361808713,LastTimestamp:2026-07-16 20:33:13.258529735 +0000 UTC m=+0.361808713,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-200.ec2.internal,}"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.272405    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.272644    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273026    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273029    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273055    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273165    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273174    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 20:33:13.406163 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273743    2580 factory.go:55] Registering systemd factory
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273759    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273956    2580 factory.go:153] Registering CRI-O factory
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.273968    2580 factory.go:223] Registration of the crio container factory successfully
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.274032    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.274051    2580 factory.go:103] Registering Raw factory
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.274063    2580 manager.go:1196] Started watching for new ooms in manager
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.274412    2580 manager.go:349] Starting recovery of all containers
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.274418    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.274737    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.309198    2580 manager.go:354] Recovery completed
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.313288    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.316856    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.316890    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.316902    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.317474    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.317489    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.317552    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.317586    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.318778    2580 policy_none.go:49] "None policy: Start"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.318799    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.318813    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.318910    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.318936    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.318954    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.319996    2580 policy_none.go:47] "Start"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.371277    2580 manager.go:337] "Starting Device Plugin manager"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.371314    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.371327    2580 server.go:87] "Starting device plugin registration server"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.371675    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.371688    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.371809    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.371904    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.371913    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.372353    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 20:33:13.408060 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.372407    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.478556    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.479744    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.479775    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.479792    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.479823    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.483424    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-200.ec2.internal\" not found" node="ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.488339    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.488367    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-200.ec2.internal\": node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.490397    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.492031    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.492070    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.492093    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.492135    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.494311    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:33:13.516507 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.502749    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.600874    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"]
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.600963    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.602834    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.602885    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.602914    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.602926    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.604146    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.604295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.604329    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.605013    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.605022    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.605039    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.605044    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.605050    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.605054    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.606266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.606288    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.606953    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.606977    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:33:13.638634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.606987    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:33:13.742381 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.664511    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-200.ec2.internal\" not found" node="ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.742381 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.669630    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-200.ec2.internal\" not found" node="ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.742381 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.679308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b30ae5bc27eeb039f1a3250327b5a6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-200.ec2.internal\" (UID: \"a0b30ae5bc27eeb039f1a3250327b5a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.742381 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.679334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.742381 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.679354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.742381 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.715948    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:13.908513 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.780456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.908513 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.780493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.908513 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.780513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b30ae5bc27eeb039f1a3250327b5a6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-200.ec2.internal\" (UID: \"a0b30ae5bc27eeb039f1a3250327b5a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.908513 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.780575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b30ae5bc27eeb039f1a3250327b5a6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-200.ec2.internal\" (UID: \"a0b30ae5bc27eeb039f1a3250327b5a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.908513 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.780601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.908513 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.780585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:13.908513 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.817143    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:13.918089 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:13.917784    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:14.064646 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.971001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:14.064646 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:13.973172    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:14.064646 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:14.018563    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:14.237433 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:14.121278    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:14.237433 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.210894    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 20:33:14.237433 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.211086    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 20:33:14.237433 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:14.221983    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:14.423553 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.273841    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 20:28:13 +0000 UTC" deadline="2028-03-13 07:19:16.740879221 +0000 UTC"
Jul 16 20:33:14.423553 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.273878    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14530h46m2.467005322s"
Jul 16 20:33:14.423553 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:14.324164    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:14.582456 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:14.441921    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 16 20:33:14.582456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.452749    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:33:14.582456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.477677    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:14.582456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.489830    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 20:33:14.582456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.498529    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 16 20:33:14.582456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.506629    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 20:33:14.730680 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.644913    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:33:14.730680 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:14.668977    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2f093a36821ff3dcaa0f49fa074849.slice/crio-e4c9b210f46f1aa41963dbf1f308eccc33313002afeffeeba114a6bb073fc1f7 WatchSource:0}: Error finding container e4c9b210f46f1aa41963dbf1f308eccc33313002afeffeeba114a6bb073fc1f7: Status 404 returned error can't find the container with id e4c9b210f46f1aa41963dbf1f308eccc33313002afeffeeba114a6bb073fc1f7
Jul 16 20:33:14.730680 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:14.669681    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b30ae5bc27eeb039f1a3250327b5a6.slice/crio-48d77682556b87e12d4dd28d564c61b8985d4975af7b34b74b8e2d1bf522562b WatchSource:0}: Error finding container 48d77682556b87e12d4dd28d564c61b8985d4975af7b34b74b8e2d1bf522562b: Status 404 returned error can't find the container with id 48d77682556b87e12d4dd28d564c61b8985d4975af7b34b74b8e2d1bf522562b
Jul 16 20:33:14.730680 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.676765    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:33:14.836584 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:14.836481    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:33:15.249336 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.249245    2580 apiserver.go:52] "Watching apiserver"
Jul 16 20:33:15.256462 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.256433    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 20:33:15.257146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.257094    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-dlcwv","openshift-multus/network-metrics-daemon-c4m8f","openshift-network-diagnostics/network-check-target-z4nkd","openshift-network-operator/iptables-alerter-bdtcg","openshift-ovn-kubernetes/ovnkube-node-2qszx","kube-system/global-pull-secret-syncer-dv7n5","kube-system/konnectivity-agent-qxzj8","kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal","openshift-dns/node-resolver-x2tl9","openshift-image-registry/node-ca-rnthw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal","openshift-multus/multus-additional-cni-plugins-sr7vx","openshift-multus/multus-zkwks","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"]
Jul 16 20:33:15.258762 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.258737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:15.260494 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.260450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.261209 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.261189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:15.261292 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.261260    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:15.261976 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.261819    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 20:33:15.262071 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.262038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8pqsb\""
Jul 16 20:33:15.262245 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.262218    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 20:33:15.262449 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.262432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.263563 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.263540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.266742 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.266718    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 20:33:15.267020 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.267001    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jrftb\""
Jul 16 20:33:15.267215 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.267199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 20:33:15.267441 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.267424    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 20:33:15.267635 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.267617    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 20:33:15.267946 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.267928    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 20:33:15.268179 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.268161    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 20:33:15.268240 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.268220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l55tb\""
Jul 16 20:33:15.268318 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.268166    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 20:33:15.268432 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.268417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.268501 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.268479    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:15.268561 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.268517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.268624 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.268564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.269295 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.269256    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 20:33:15.269580 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.269559    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 20:33:15.269676 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.269609    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 20:33:15.269903 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.269883    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 20:33:15.270016 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.270000    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 20:33:15.270143 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.270086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jppmm\""
Jul 16 20:33:15.270213 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.270185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 20:33:15.271182 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.271162    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:15.271327 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.271296    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:15.272739 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.272717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.272899 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.272880    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 20:33:15.273715 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.273409    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 20:33:15.273715 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.273529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r7drx\""
Jul 16 20:33:15.273715 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.273548    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 20:33:15.278622 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.274883    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 20:33:15.278622 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.275521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.278622 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.276378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nffj2\""
Jul 16 20:33:15.278622 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.278580    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 20:33:15.279177 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.278966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fzshc\""
Jul 16 20:33:15.279245 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.279187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 20:33:15.280624 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.279335    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 20:33:15.280624 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.279703    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 20:33:15.281844 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.281824    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 20:33:15.283611 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.282612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.283611 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.282954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7shrg\""
Jul 16 20:33:15.283611 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.283324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 20:33:15.283834 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.283657    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.286697    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.287338    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.287674    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-865jb\""
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.289908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-kubelet\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.289944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-cni-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-hostroot\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9eefcd28-31b3-49ae-a706-af9e9efdb810-iptables-alerter-script\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9eefcd28-31b3-49ae-a706-af9e9efdb810-host-slash\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-log-socket\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-cni-bin\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-k8s-cni-cncf-io\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.290396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-conf-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-dbus\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e25e247f-b941-4c58-9bf0-61fb3ca27019-agent-certs\") pod \"konnectivity-agent-qxzj8\" (UID: \"e25e247f-b941-4c58-9bf0-61fb3ca27019\") " pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-kubernetes\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-run\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-cnibin\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-cni-bin\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-registration-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-node-log\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-tuned\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-socket-dir-parent\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.290984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-systemd\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.291312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-sys\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-kubelet\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-kubelet-config\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-slash\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-var-lib-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb1313d4-cea7-4463-bf77-c416f3f4a096-cni-binary-copy\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-etc-kubernetes\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93410d88-85b1-4b03-b66b-d6de95740d86-hosts-file\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-run-netns\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-systemd\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvs7\" (UniqueName: \"kubernetes.io/projected/a839f97a-d4f3-4093-8c47-1ab868fce6f9-kube-api-access-csvs7\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-modprobe-d\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysconfig\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-system-cni-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-sys-fs\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.292146 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovnkube-config\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovn-node-metrics-cert\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5dl\" (UniqueName: \"kubernetes.io/projected/875688ee-e4d0-4278-a9bb-7587e6374c47-kube-api-access-2l5dl\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-netns\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/875688ee-e4d0-4278-a9bb-7587e6374c47-tmp\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-socket-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-env-overrides\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-daemon-config\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.291994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-multus-certs\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93410d88-85b1-4b03-b66b-d6de95740d86-tmp-dir\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-device-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovnkube-script-lib\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-lib-modules\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-systemd-units\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-etc-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysctl-conf\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.292998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-var-lib-kubelet\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-host\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-serviceca\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kpw\" (UniqueName: \"kubernetes.io/projected/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-kube-api-access-n7kpw\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdnz\" (UniqueName: \"kubernetes.io/projected/8266a436-acb6-4c37-82be-9cccbbb1e9bc-kube-api-access-qqdnz\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-cni-multus\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292832    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.292531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t96r9\" (UniqueName: \"kubernetes.io/projected/9eefcd28-31b3-49ae-a706-af9e9efdb810-kube-api-access-t96r9\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-ovn\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-cni-netd\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-os-release\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h84s\" (UniqueName: \"kubernetes.io/projected/cb1313d4-cea7-4463-bf77-c416f3f4a096-kube-api-access-2h84s\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljh9c\" (UniqueName: \"kubernetes.io/projected/5c121780-db69-4045-b1b5-399526502e8b-kube-api-access-ljh9c\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysctl-d\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-host\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.293884 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ckh\" (UniqueName: \"kubernetes.io/projected/93410d88-85b1-4b03-b66b-d6de95740d86-kube-api-access-q4ckh\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.295077 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.293436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e25e247f-b941-4c58-9bf0-61fb3ca27019-konnectivity-ca\") pod \"konnectivity-agent-qxzj8\" (UID: \"e25e247f-b941-4c58-9bf0-61fb3ca27019\") " pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:15.312244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.312208    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ppsxg"
Jul 16 20:33:15.320697 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.320677    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ppsxg"
Jul 16 20:33:15.374289 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.374257    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 20:33:15.394639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t96r9\" (UniqueName: \"kubernetes.io/projected/9eefcd28-31b3-49ae-a706-af9e9efdb810-kube-api-access-t96r9\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.394800 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-ovn\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.394800 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-cni-netd\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.394800 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-os-release\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.394800 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h84s\" (UniqueName: \"kubernetes.io/projected/cb1313d4-cea7-4463-bf77-c416f3f4a096-kube-api-access-2h84s\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.394800 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljh9c\" (UniqueName: \"kubernetes.io/projected/5c121780-db69-4045-b1b5-399526502e8b-kube-api-access-ljh9c\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.394800 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysctl-d\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-host\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ckh\" (UniqueName: \"kubernetes.io/projected/93410d88-85b1-4b03-b66b-d6de95740d86-kube-api-access-q4ckh\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e25e247f-b941-4c58-9bf0-61fb3ca27019-konnectivity-ca\") pod \"konnectivity-agent-qxzj8\" (UID: \"e25e247f-b941-4c58-9bf0-61fb3ca27019\") " pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-kubelet\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-cni-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-hostroot\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-system-cni-dir\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.394984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9eefcd28-31b3-49ae-a706-af9e9efdb810-iptables-alerter-script\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9eefcd28-31b3-49ae-a706-af9e9efdb810-host-slash\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-log-socket\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.395116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-kubelet\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-ovn\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-cni-netd\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysctl-d\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-cni-bin\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-cni-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-os-release\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-k8s-cni-cncf-io\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-conf-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-dbus\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-host\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e25e247f-b941-4c58-9bf0-61fb3ca27019-agent-certs\") pod \"konnectivity-agent-qxzj8\" (UID: \"e25e247f-b941-4c58-9bf0-61fb3ca27019\") " pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-kubernetes\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-run\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-cnibin\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-cni-bin\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.395763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-registration-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-node-log\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9eefcd28-31b3-49ae-a706-af9e9efdb810-iptables-alerter-script\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.395972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-tuned\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-hostroot\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-socket-dir-parent\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cnibin\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.396117    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-systemd\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-sys\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.396189    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs podName:8266a436-acb6-4c37-82be-9cccbbb1e9bc nodeName:}" failed. No retries permitted until 2026-07-16 20:33:15.896167388 +0000 UTC m=+2.999446359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs") pod "network-metrics-daemon-c4m8f" (UID: "8266a436-acb6-4c37-82be-9cccbbb1e9bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-kubelet\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e25e247f-b941-4c58-9bf0-61fb3ca27019-konnectivity-ca\") pod \"konnectivity-agent-qxzj8\" (UID: \"e25e247f-b941-4c58-9bf0-61fb3ca27019\") " pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:15.396702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-kubelet-config\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-cni-bin\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-slash\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-k8s-cni-cncf-io\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-var-lib-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-socket-dir-parent\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb1313d4-cea7-4463-bf77-c416f3f4a096-cni-binary-copy\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-conf-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9eefcd28-31b3-49ae-a706-af9e9efdb810-host-slash\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-etc-kubernetes\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-log-socket\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93410d88-85b1-4b03-b66b-d6de95740d86-hosts-file\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-dbus\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-run-netns\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-systemd\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.397554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csvs7\" (UniqueName: \"kubernetes.io/projected/a839f97a-d4f3-4093-8c47-1ab868fce6f9-kube-api-access-csvs7\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-modprobe-d\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysconfig\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-system-cni-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-sys-fs\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-systemd\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovnkube-config\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovn-node-metrics-cert\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-sys\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5dl\" (UniqueName: \"kubernetes.io/projected/875688ee-e4d0-4278-a9bb-7587e6374c47-kube-api-access-2l5dl\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-netns\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-kubelet\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/875688ee-e4d0-4278-a9bb-7587e6374c47-tmp\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-kubelet-config\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-socket-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-cni-bin\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.398443 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-os-release\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-env-overrides\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-daemon-config\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-kubernetes\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-multus-certs\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93410d88-85b1-4b03-b66b-d6de95740d86-tmp-dir\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-run\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-device-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovnkube-script-lib\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-cnibin\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.397092    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-lib-modules\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmpp\" (UniqueName: \"kubernetes.io/projected/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-kube-api-access-kxmpp\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-systemd-units\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-registration-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-etc-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.399292 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysctl-conf\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-node-log\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovnkube-config\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-env-overrides\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-slash\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-var-lib-kubelet\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-host\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysctl-conf\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-serviceca\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kpw\" (UniqueName: \"kubernetes.io/projected/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-kube-api-access-n7kpw\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdnz\" (UniqueName: \"kubernetes.io/projected/8266a436-acb6-4c37-82be-9cccbbb1e9bc-kube-api-access-qqdnz\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-cni-multus\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.397994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-var-lib-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb1313d4-cea7-4463-bf77-c416f3f4a096-multus-daemon-config\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.400175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-multus-certs\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-serviceca\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.396742    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb1313d4-cea7-4463-bf77-c416f3f4a096-cni-binary-copy\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-etc-kubernetes\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93410d88-85b1-4b03-b66b-d6de95740d86-hosts-file\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-host-run-netns\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-systemd\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-var-lib-kubelet\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.398951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-modprobe-d\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-sysconfig\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-system-cni-dir\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-socket-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-sys-fs\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-lib-modules\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-run-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93410d88-85b1-4b03-b66b-d6de95740d86-tmp-dir\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.401015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c121780-db69-4045-b1b5-399526502e8b-device-dir\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.401909 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.399611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-var-lib-cni-multus\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.401909 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.400087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovnkube-script-lib\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.401909 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.400371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb1313d4-cea7-4463-bf77-c416f3f4a096-host-run-netns\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.401909 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.400606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/875688ee-e4d0-4278-a9bb-7587e6374c47-host\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.401909 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.400624    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret podName:783a85c6-76a0-49bf-99fc-b1083bdc1bf7 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:15.90060531 +0000 UTC m=+3.003884280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret") pod "global-pull-secret-syncer-dv7n5" (UID: "783a85c6-76a0-49bf-99fc-b1083bdc1bf7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:15.401909 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.400668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-systemd-units\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.401909 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.400683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a839f97a-d4f3-4093-8c47-1ab868fce6f9-etc-openvswitch\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.402793 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.402456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/875688ee-e4d0-4278-a9bb-7587e6374c47-etc-tuned\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.402793 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.402581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a839f97a-d4f3-4093-8c47-1ab868fce6f9-ovn-node-metrics-cert\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.404670 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.403274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e25e247f-b941-4c58-9bf0-61fb3ca27019-agent-certs\") pod \"konnectivity-agent-qxzj8\" (UID: \"e25e247f-b941-4c58-9bf0-61fb3ca27019\") " pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:15.404670 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.404329    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:15.404670 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.404352    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:15.404670 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.404366    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf6bx for pod openshift-network-diagnostics/network-check-target-z4nkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:15.404670 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.404417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/875688ee-e4d0-4278-a9bb-7587e6374c47-tmp\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.404670 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.404456    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx podName:cf97482e-a61d-4ccd-a005-318f050f2238 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:15.904437339 +0000 UTC m=+3.007716326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lf6bx" (UniqueName: "kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx") pod "network-check-target-z4nkd" (UID: "cf97482e-a61d-4ccd-a005-318f050f2238") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:15.405454 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.405432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljh9c\" (UniqueName: \"kubernetes.io/projected/5c121780-db69-4045-b1b5-399526502e8b-kube-api-access-ljh9c\") pod \"aws-ebs-csi-driver-node-zpxd9\" (UID: \"5c121780-db69-4045-b1b5-399526502e8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.410653 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.409642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h84s\" (UniqueName: \"kubernetes.io/projected/cb1313d4-cea7-4463-bf77-c416f3f4a096-kube-api-access-2h84s\") pod \"multus-zkwks\" (UID: \"cb1313d4-cea7-4463-bf77-c416f3f4a096\") " pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.410653 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.409628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t96r9\" (UniqueName: \"kubernetes.io/projected/9eefcd28-31b3-49ae-a706-af9e9efdb810-kube-api-access-t96r9\") pod \"iptables-alerter-bdtcg\" (UID: \"9eefcd28-31b3-49ae-a706-af9e9efdb810\") " pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.410653 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.409961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ckh\" (UniqueName: \"kubernetes.io/projected/93410d88-85b1-4b03-b66b-d6de95740d86-kube-api-access-q4ckh\") pod \"node-resolver-x2tl9\" (UID: \"93410d88-85b1-4b03-b66b-d6de95740d86\") " pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.410653 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.409987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdnz\" (UniqueName: \"kubernetes.io/projected/8266a436-acb6-4c37-82be-9cccbbb1e9bc-kube-api-access-qqdnz\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:15.410653 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.410505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kpw\" (UniqueName: \"kubernetes.io/projected/24a3af80-d268-4bb1-8a80-b6a69d5d6db2-kube-api-access-n7kpw\") pod \"node-ca-rnthw\" (UID: \"24a3af80-d268-4bb1-8a80-b6a69d5d6db2\") " pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.410977 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.410872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvs7\" (UniqueName: \"kubernetes.io/projected/a839f97a-d4f3-4093-8c47-1ab868fce6f9-kube-api-access-csvs7\") pod \"ovnkube-node-2qszx\" (UID: \"a839f97a-d4f3-4093-8c47-1ab868fce6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.412638 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.412574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5dl\" (UniqueName: \"kubernetes.io/projected/875688ee-e4d0-4278-a9bb-7587e6374c47-kube-api-access-2l5dl\") pod \"tuned-dlcwv\" (UID: \"875688ee-e4d0-4278-a9bb-7587e6374c47\") " pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.497957 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.497861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal" event={"ID":"a0b30ae5bc27eeb039f1a3250327b5a6","Type":"ContainerStarted","Data":"48d77682556b87e12d4dd28d564c61b8985d4975af7b34b74b8e2d1bf522562b"}
Jul 16 20:33:15.498456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-system-cni-dir\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498631 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-system-cni-dir\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498631 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cnibin\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498631 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cnibin\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498631 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498631 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498631 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-os-release\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498937 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmpp\" (UniqueName: \"kubernetes.io/projected/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-kube-api-access-kxmpp\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498937 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.498937 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.498887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.499134 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.499113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-os-release\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.499577 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.499481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.499577 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.499481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.499734 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.499660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal" event={"ID":"9e2f093a36821ff3dcaa0f49fa074849","Type":"ContainerStarted","Data":"e4c9b210f46f1aa41963dbf1f308eccc33313002afeffeeba114a6bb073fc1f7"}
Jul 16 20:33:15.499734 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.499669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.508509 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.508479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmpp\" (UniqueName: \"kubernetes.io/projected/230c5d01-b3a9-4a57-a642-4ac6905b4b4b-kube-api-access-kxmpp\") pod \"multus-additional-cni-plugins-sr7vx\" (UID: \"230c5d01-b3a9-4a57-a642-4ac6905b4b4b\") " pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.581834 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.581779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:15.597394 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.597368    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zkwks"
Jul 16 20:33:15.612705 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.612675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdtcg"
Jul 16 20:33:15.617009 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.616988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:15.625211 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.625189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlcwv"
Jul 16 20:33:15.633832 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.633804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x2tl9"
Jul 16 20:33:15.644209 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.644182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rnthw"
Jul 16 20:33:15.651626 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.651600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9"
Jul 16 20:33:15.656254 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.656230    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:33:15.658506 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.658485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sr7vx"
Jul 16 20:33:15.902584 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.902473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:15.902584 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:15.902533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:15.902819 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.902678    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:15.902819 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.902763    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs podName:8266a436-acb6-4c37-82be-9cccbbb1e9bc nodeName:}" failed. No retries permitted until 2026-07-16 20:33:16.90274575 +0000 UTC m=+4.006024714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs") pod "network-metrics-daemon-c4m8f" (UID: "8266a436-acb6-4c37-82be-9cccbbb1e9bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:15.902819 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.902688    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:15.902992 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:15.902862    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret podName:783a85c6-76a0-49bf-99fc-b1083bdc1bf7 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:16.902840922 +0000 UTC m=+4.006119891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret") pod "global-pull-secret-syncer-dv7n5" (UID: "783a85c6-76a0-49bf-99fc-b1083bdc1bf7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:16.004070 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:16.004021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:16.004271 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.004180    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:16.004271 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.004197    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:16.004271 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.004207    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf6bx for pod openshift-network-diagnostics/network-check-target-z4nkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:16.004271 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.004265    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx podName:cf97482e-a61d-4ccd-a005-318f050f2238 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:17.004248039 +0000 UTC m=+4.107527009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf6bx" (UniqueName: "kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx") pod "network-check-target-z4nkd" (UID: "cf97482e-a61d-4ccd-a005-318f050f2238") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:16.323175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:16.323089    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 20:28:15 +0000 UTC" deadline="2027-12-12 03:21:57.006248807 +0000 UTC"
Jul 16 20:33:16.323175 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:16.323125    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12318h48m40.683127603s"
Jul 16 20:33:16.493371 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:16.493333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:16.493371 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:16.493362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:16.493631 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.493459    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:16.493631 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.493592    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:16.666252 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:16.666224    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875688ee_e4d0_4278_a9bb_7587e6374c47.slice/crio-2498c1ea313501f359ee4ef8b45b7341ec42b62e122c152ae7d82afe1bef1f5b WatchSource:0}: Error finding container 2498c1ea313501f359ee4ef8b45b7341ec42b62e122c152ae7d82afe1bef1f5b: Status 404 returned error can't find the container with id 2498c1ea313501f359ee4ef8b45b7341ec42b62e122c152ae7d82afe1bef1f5b
Jul 16 20:33:16.667933 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:16.667911    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda839f97a_d4f3_4093_8c47_1ab868fce6f9.slice/crio-8970f5e98a646a8d25d15c14f6a6ab4a2fed76889109c165c317796589f788bc WatchSource:0}: Error finding container 8970f5e98a646a8d25d15c14f6a6ab4a2fed76889109c165c317796589f788bc: Status 404 returned error can't find the container with id 8970f5e98a646a8d25d15c14f6a6ab4a2fed76889109c165c317796589f788bc
Jul 16 20:33:16.912492 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:16.912422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:16.912492 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:16.912469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:16.912724 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.912616    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:16.912724 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.912647    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:16.912724 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.912676    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs podName:8266a436-acb6-4c37-82be-9cccbbb1e9bc nodeName:}" failed. No retries permitted until 2026-07-16 20:33:18.912660813 +0000 UTC m=+6.015939778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs") pod "network-metrics-daemon-c4m8f" (UID: "8266a436-acb6-4c37-82be-9cccbbb1e9bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:16.912724 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:16.912702    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret podName:783a85c6-76a0-49bf-99fc-b1083bdc1bf7 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:18.912684325 +0000 UTC m=+6.015963305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret") pod "global-pull-secret-syncer-dv7n5" (UID: "783a85c6-76a0-49bf-99fc-b1083bdc1bf7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:17.013982 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.013766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:17.014128 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:17.013936    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:17.014128 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:17.014048    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:17.014128 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:17.014061    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf6bx for pod openshift-network-diagnostics/network-check-target-z4nkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:17.014128 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:17.014114    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx podName:cf97482e-a61d-4ccd-a005-318f050f2238 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:19.014098889 +0000 UTC m=+6.117377874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf6bx" (UniqueName: "kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx") pod "network-check-target-z4nkd" (UID: "cf97482e-a61d-4ccd-a005-318f050f2238") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:17.323500 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.323342    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 20:28:15 +0000 UTC" deadline="2028-01-01 11:29:58.539307652 +0000 UTC"
Jul 16 20:33:17.323500 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.323410    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12806h56m41.215902831s"
Jul 16 20:33:17.496018 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.495989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:17.496210 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:17.496122    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:17.512203 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.512163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qxzj8" event={"ID":"e25e247f-b941-4c58-9bf0-61fb3ca27019","Type":"ContainerStarted","Data":"ade4e2d8c667151e9ef5c539af37e7376affbba98e00d8d7076a4564c99bbd67"}
Jul 16 20:33:17.522333 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.522292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zkwks" event={"ID":"cb1313d4-cea7-4463-bf77-c416f3f4a096","Type":"ContainerStarted","Data":"c4618895afd668fd0ef789fbf78159ac4f5c1a6e321dca252c873d17c4f86507"}
Jul 16 20:33:17.529256 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.529137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdtcg" event={"ID":"9eefcd28-31b3-49ae-a706-af9e9efdb810","Type":"ContainerStarted","Data":"8825470d77623c5ec10883ffb90933a8d4040fb4cd8d2ac67a8bd68a3598140b"}
Jul 16 20:33:17.535227 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.535191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"8970f5e98a646a8d25d15c14f6a6ab4a2fed76889109c165c317796589f788bc"}
Jul 16 20:33:17.539171 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.539136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlcwv" event={"ID":"875688ee-e4d0-4278-a9bb-7587e6374c47","Type":"ContainerStarted","Data":"2498c1ea313501f359ee4ef8b45b7341ec42b62e122c152ae7d82afe1bef1f5b"}
Jul 16 20:33:17.546184 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.546149    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e2f093a36821ff3dcaa0f49fa074849" containerID="164e4a865831a616bb278a8e8fee44ed5b3f77b09963b958c66f9b8c7c4ff8b2" exitCode=0
Jul 16 20:33:17.546318 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.546243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal" event={"ID":"9e2f093a36821ff3dcaa0f49fa074849","Type":"ContainerDied","Data":"164e4a865831a616bb278a8e8fee44ed5b3f77b09963b958c66f9b8c7c4ff8b2"}
Jul 16 20:33:17.563213 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.563175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x2tl9" event={"ID":"93410d88-85b1-4b03-b66b-d6de95740d86","Type":"ContainerStarted","Data":"e87acf50f0972c43cb775361c05d369f0dc1f84a85fd699ef06ae5eec77870fd"}
Jul 16 20:33:17.588297 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.588263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rnthw" event={"ID":"24a3af80-d268-4bb1-8a80-b6a69d5d6db2","Type":"ContainerStarted","Data":"2dfaf764c8ceff9bfe6f6afb73b12e6fc43eaf7310284ba52d59a9a7581bda3c"}
Jul 16 20:33:17.606210 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.606171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal" event={"ID":"a0b30ae5bc27eeb039f1a3250327b5a6","Type":"ContainerStarted","Data":"8e3aaa9f7a4191d7628058840465a5cbce6855c381a20d76a0f250d28460d5c2"}
Jul 16 20:33:17.613395 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.613361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerStarted","Data":"41d29bb16fc50872c84f27a8a2fc92e24a00a03bb4551353333eff43a8f94300"}
Jul 16 20:33:17.615677 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:17.615650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9" event={"ID":"5c121780-db69-4045-b1b5-399526502e8b","Type":"ContainerStarted","Data":"749ace0977b7284d49037cf2c8656704ebb55ac7134f7335c715d2283c7e303b"}
Jul 16 20:33:18.493097 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:18.493059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:18.493718 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:18.493204    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:18.493718 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:18.493297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:18.493718 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:18.493384    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:18.642083 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:18.641548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal" event={"ID":"9e2f093a36821ff3dcaa0f49fa074849","Type":"ContainerStarted","Data":"f2fb576fed157fd92e873006e2c6cbede546ef77d30aac5dc0ff7e2cbd203e38"}
Jul 16 20:33:18.666242 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:18.665603    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal" podStartSLOduration=3.665565393 podStartE2EDuration="3.665565393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:17.634679259 +0000 UTC m=+4.737958261" watchObservedRunningTime="2026-07-16 20:33:18.665565393 +0000 UTC m=+5.768844382"
Jul 16 20:33:18.933095 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:18.933005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:18.933095 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:18.933060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:18.933337 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:18.933169    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:18.933337 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:18.933237    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret podName:783a85c6-76a0-49bf-99fc-b1083bdc1bf7 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:22.933216575 +0000 UTC m=+10.036495556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret") pod "global-pull-secret-syncer-dv7n5" (UID: "783a85c6-76a0-49bf-99fc-b1083bdc1bf7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:18.933658 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:18.933169    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:18.933658 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:18.933595    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs podName:8266a436-acb6-4c37-82be-9cccbbb1e9bc nodeName:}" failed. No retries permitted until 2026-07-16 20:33:22.933546771 +0000 UTC m=+10.036825753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs") pod "network-metrics-daemon-c4m8f" (UID: "8266a436-acb6-4c37-82be-9cccbbb1e9bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:19.034269 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:19.034222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:19.034486 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:19.034466    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:19.034566 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:19.034498    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:19.034566 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:19.034512    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf6bx for pod openshift-network-diagnostics/network-check-target-z4nkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:19.034699 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:19.034594    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx podName:cf97482e-a61d-4ccd-a005-318f050f2238 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:23.034555577 +0000 UTC m=+10.137834561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf6bx" (UniqueName: "kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx") pod "network-check-target-z4nkd" (UID: "cf97482e-a61d-4ccd-a005-318f050f2238") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:19.494298 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:19.493784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:19.494298 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:19.493931    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:20.492726 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:20.492690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:20.492931 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:20.492829    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:20.493253 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:20.493230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:20.493376 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:20.493353    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:21.495808 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:21.495266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:21.495808 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:21.495403    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:22.493103 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:22.493067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:22.493273 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:22.493189    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:22.493350 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:22.493285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:22.493444 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:22.493418    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:22.970242 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:22.970200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:22.970721 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:22.970255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:22.970721 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:22.970375    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:22.970721 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:22.970386    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:22.970721 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:22.970445    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret podName:783a85c6-76a0-49bf-99fc-b1083bdc1bf7 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:30.970425868 +0000 UTC m=+18.073704853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret") pod "global-pull-secret-syncer-dv7n5" (UID: "783a85c6-76a0-49bf-99fc-b1083bdc1bf7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:22.970721 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:22.970464    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs podName:8266a436-acb6-4c37-82be-9cccbbb1e9bc nodeName:}" failed. No retries permitted until 2026-07-16 20:33:30.970454671 +0000 UTC m=+18.073733640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs") pod "network-metrics-daemon-c4m8f" (UID: "8266a436-acb6-4c37-82be-9cccbbb1e9bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:23.070924 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:23.070884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:23.071099 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:23.071078    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:23.071177 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:23.071108    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:23.071177 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:23.071121    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf6bx for pod openshift-network-diagnostics/network-check-target-z4nkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:23.071285 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:23.071188    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx podName:cf97482e-a61d-4ccd-a005-318f050f2238 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:31.071167902 +0000 UTC m=+18.174446887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf6bx" (UniqueName: "kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx") pod "network-check-target-z4nkd" (UID: "cf97482e-a61d-4ccd-a005-318f050f2238") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:23.494495 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:23.494111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:23.494495 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:23.494232    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:24.492596 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:24.492279    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:24.492596 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:24.492302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:24.492596 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:24.492410    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:24.492596 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:24.492558    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:25.492671 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:25.492482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:25.492671 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:25.492612    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:26.492477 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:26.492438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:26.492670 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:26.492458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:26.492670 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:26.492585    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:26.493045 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:26.492703    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:27.492337 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:27.492309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:27.492488 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:27.492416    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:28.493260 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:28.493229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:28.493735 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:28.493229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:28.493735 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:28.493355    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:28.493735 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:28.493413    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:29.493396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:29.493359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:29.493871 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:29.493501    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:30.492861 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:30.492821    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:30.492861 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:30.492862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:30.493120 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:30.492943    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:30.493120 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:30.493091    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:31.029182 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:31.029142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:31.029656 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:31.029200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:31.029656 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.029314    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:31.029656 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.029377    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret podName:783a85c6-76a0-49bf-99fc-b1083bdc1bf7 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:47.029358959 +0000 UTC m=+34.132637924 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret") pod "global-pull-secret-syncer-dv7n5" (UID: "783a85c6-76a0-49bf-99fc-b1083bdc1bf7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:31.029656 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.029402    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:31.029656 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.029475    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs podName:8266a436-acb6-4c37-82be-9cccbbb1e9bc nodeName:}" failed. No retries permitted until 2026-07-16 20:33:47.02945467 +0000 UTC m=+34.132733646 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs") pod "network-metrics-daemon-c4m8f" (UID: "8266a436-acb6-4c37-82be-9cccbbb1e9bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:31.130422 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:31.130382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:31.130639 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.130576    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:31.130639 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.130598    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:31.130639 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.130610    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf6bx for pod openshift-network-diagnostics/network-check-target-z4nkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:31.130789 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.130675    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx podName:cf97482e-a61d-4ccd-a005-318f050f2238 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:47.130656298 +0000 UTC m=+34.233935275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf6bx" (UniqueName: "kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx") pod "network-check-target-z4nkd" (UID: "cf97482e-a61d-4ccd-a005-318f050f2238") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:31.492912 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:31.492877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:31.493073 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:31.493021    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:32.493150 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:32.493111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:32.493605 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:32.493111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:32.493605 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:32.493255    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:32.493605 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:32.493318    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:33.493110 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:33.493074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:33.493300 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:33.493185    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:34.492472 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:34.492441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:34.492655 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:34.492449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:34.492655 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:34.492577    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:34.492655 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:34.492622    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:35.493298 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:35.493266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:35.493773 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:35.493370    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:36.492566 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.492305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:36.492767 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.492326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:36.492767 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:36.492692    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:36.492767 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:36.492753    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:36.680047 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.680012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlcwv" event={"ID":"875688ee-e4d0-4278-a9bb-7587e6374c47","Type":"ContainerStarted","Data":"c0b0bcad1465afcd4707d2576474a3d210fd8413af32b9c52676bd5ca062d6c2"}
Jul 16 20:33:36.681565 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.681533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x2tl9" event={"ID":"93410d88-85b1-4b03-b66b-d6de95740d86","Type":"ContainerStarted","Data":"8f281ff006851e1485dee7e8a355a09aaff19020c3feaeb756008c702f2e0d90"}
Jul 16 20:33:36.682981 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.682930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rnthw" event={"ID":"24a3af80-d268-4bb1-8a80-b6a69d5d6db2","Type":"ContainerStarted","Data":"7811764256abcb02f10a14642dd3877b93944164eff6c278237a1cc983418ade"}
Jul 16 20:33:36.684723 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.684567    2580 generic.go:356] "Generic (PLEG): container finished" podID="230c5d01-b3a9-4a57-a642-4ac6905b4b4b" containerID="509c6448b78fd6d3b7640d4307980de5e38f5e190bd519330ddbe5dcf05accbd" exitCode=0
Jul 16 20:33:36.684723 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.684699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerDied","Data":"509c6448b78fd6d3b7640d4307980de5e38f5e190bd519330ddbe5dcf05accbd"}
Jul 16 20:33:36.686358 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.686324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9" event={"ID":"5c121780-db69-4045-b1b5-399526502e8b","Type":"ContainerStarted","Data":"f63bcbd12aeb6f137d550d0c8d1c4dbd7bf6462f5d6fe6cc320d41f5a6b5784b"}
Jul 16 20:33:36.687860 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.687835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qxzj8" event={"ID":"e25e247f-b941-4c58-9bf0-61fb3ca27019","Type":"ContainerStarted","Data":"c843f52f23eedf6a0f2a2bfcbe46daa0e5738849a6006fb5d11ca8e9de19cf64"}
Jul 16 20:33:36.689951 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.689909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zkwks" event={"ID":"cb1313d4-cea7-4463-bf77-c416f3f4a096","Type":"ContainerStarted","Data":"551f7441ad6f62ab82807c8c97dc0f3c11feca2f20ba289bb0eaa95f6015367d"}
Jul 16 20:33:36.692114 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.692072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"58dd7ded4024796fce6acd5835e05d0768d4474f6068c0d0544b661180208738"}
Jul 16 20:33:36.692220 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.692149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"abb0b5a576344c6b599219dd9af5661260022b6e1c78cf9d4ba68d7218335a64"}
Jul 16 20:33:36.700343 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.700296    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal" podStartSLOduration=21.700282871 podStartE2EDuration="21.700282871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:18.666228195 +0000 UTC m=+5.769507185" watchObservedRunningTime="2026-07-16 20:33:36.700282871 +0000 UTC m=+23.803561856"
Jul 16 20:33:36.700641 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.700604    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dlcwv" podStartSLOduration=3.355075721 podStartE2EDuration="22.700595379s" totalImagesPullingTime="19.345519658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:36.700065287 +0000 UTC m=+23.803344298" watchObservedRunningTime="2026-07-16 20:33:36.700595379 +0000 UTC m=+23.803874366"
Jul 16 20:33:36.716079 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.716030    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x2tl9" podStartSLOduration=3.4117672900000002 podStartE2EDuration="22.716011064s" totalImagesPullingTime="19.304243774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:36.715344012 +0000 UTC m=+23.818622997" watchObservedRunningTime="2026-07-16 20:33:36.716011064 +0000 UTC m=+23.819290050"
Jul 16 20:33:36.736424 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.736375    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rnthw" podStartSLOduration=3.418169234 podStartE2EDuration="22.736359719s" totalImagesPullingTime="19.318190485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:36.736154102 +0000 UTC m=+23.839433115" watchObservedRunningTime="2026-07-16 20:33:36.736359719 +0000 UTC m=+23.839638705"
Jul 16 20:33:36.768566 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.768507    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qxzj8" podStartSLOduration=7.896704623 podStartE2EDuration="22.768491874s" totalImagesPullingTime="14.871787251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:36.768258175 +0000 UTC m=+23.871537159" watchObservedRunningTime="2026-07-16 20:33:36.768491874 +0000 UTC m=+23.871770858"
Jul 16 20:33:36.784591 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:36.784416    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zkwks" podStartSLOduration=3.432758668 podStartE2EDuration="22.784401302s" totalImagesPullingTime="19.351642634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:36.784006663 +0000 UTC m=+23.887285670" watchObservedRunningTime="2026-07-16 20:33:36.784401302 +0000 UTC m=+23.887680287"
Jul 16 20:33:37.458849 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.458812    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 20:33:37.493298 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.493263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:37.493495 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:37.493397    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:37.696459 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.696404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9" event={"ID":"5c121780-db69-4045-b1b5-399526502e8b","Type":"ContainerStarted","Data":"5771bdcbe0d375e6b245d72efe3186ff5cbc404b5a62175d37b3584c71f2efd0"}
Jul 16 20:33:37.697955 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.697927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdtcg" event={"ID":"9eefcd28-31b3-49ae-a706-af9e9efdb810","Type":"ContainerStarted","Data":"0f681a5cb39ad424710029f7ae9ee801e33338f4b46439999a2a1b03b75d834a"}
Jul 16 20:33:37.700915 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.700899    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:33:37.701283 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.701257    2580 generic.go:356] "Generic (PLEG): container finished" podID="a839f97a-d4f3-4093-8c47-1ab868fce6f9" containerID="58dd7ded4024796fce6acd5835e05d0768d4474f6068c0d0544b661180208738" exitCode=1
Jul 16 20:33:37.701442 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.701419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerDied","Data":"58dd7ded4024796fce6acd5835e05d0768d4474f6068c0d0544b661180208738"}
Jul 16 20:33:37.701553 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.701504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"8b49403aebb65662a533fc64f371d699993d2ba3371fcfe65b22231842ac45a0"}
Jul 16 20:33:37.701608 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.701576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"8a71289b92acc890e6c668a30d3f47d60783c0533e7e28521c359cdd4ca2642b"}
Jul 16 20:33:37.701641 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.701631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"3bb2893590b0dfcc31f2ff56ffefba6f8a455c4ab4057f95b1353725ff95ac71"}
Jul 16 20:33:37.701692 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.701678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"58546b1de693fee27f49fd19b7c9d88a2a3009353ab3dc18c72adadb23463820"}
Jul 16 20:33:37.711277 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:37.711236    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bdtcg" podStartSLOduration=4.374153589 podStartE2EDuration="23.711223966s" totalImagesPullingTime="19.337070377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:37.710877289 +0000 UTC m=+24.814156285" watchObservedRunningTime="2026-07-16 20:33:37.711223966 +0000 UTC m=+24.814502950"
Jul 16 20:33:38.363873 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.363838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:38.364436 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.364417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:38.388546 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.388445    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T20:33:37.458845275Z","UUID":"0441f640-1bad-434e-b180-720054d71613","Handler":null,"Name":"","Endpoint":""}
Jul 16 20:33:38.390141 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.390120    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 20:33:38.390229 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.390159    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 20:33:38.492294 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.492269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:38.492294 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.492284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:38.492492 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:38.492398    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:38.492573 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:38.492552    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:38.706113 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.706073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9" event={"ID":"5c121780-db69-4045-b1b5-399526502e8b","Type":"ContainerStarted","Data":"18a33cba9c61e7eaa1ae6b39786cfe0806efb557ff7dff481709d81d91db2746"}
Jul 16 20:33:38.706602 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.706588    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:38.706898 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.706873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qxzj8"
Jul 16 20:33:38.721458 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:38.721409    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zpxd9" podStartSLOduration=3.35664125 podStartE2EDuration="24.721394557s" totalImagesPullingTime="21.364753307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:38.721163007 +0000 UTC m=+25.824441993" watchObservedRunningTime="2026-07-16 20:33:38.721394557 +0000 UTC m=+25.824673590"
Jul 16 20:33:39.492487 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:39.492453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:39.492684 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:39.492582    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:39.711602 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:39.711565    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:33:39.712009 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:39.711981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"dcb18a5676b6a26d586c5d913aeab57c2af3364e41daa034c3462b39b4725ba2"}
Jul 16 20:33:40.492865 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:40.492667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:40.493048 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:40.492675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:40.493048 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:40.492951    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:40.493155 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:40.493045    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:41.492617 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:41.492592    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:41.493438 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:41.492712    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:41.717714 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:41.717500    2580 generic.go:356] "Generic (PLEG): container finished" podID="230c5d01-b3a9-4a57-a642-4ac6905b4b4b" containerID="c2513671ac609f257520238dbb04debfc51beebbfe8147cd68c11e00cce61b55" exitCode=0
Jul 16 20:33:41.717889 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:41.717569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerDied","Data":"c2513671ac609f257520238dbb04debfc51beebbfe8147cd68c11e00cce61b55"}
Jul 16 20:33:41.721440 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:41.721423    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:33:41.721779 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:41.721759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"bdd081caf34351477595c2bd0d7fdaa3fd4b2b5ec3d30d5bf67a5275f8ee49cc"}
Jul 16 20:33:41.722016 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:41.721990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:41.722162 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:41.722148    2580 scope.go:117] "RemoveContainer" containerID="58dd7ded4024796fce6acd5835e05d0768d4474f6068c0d0544b661180208738"
Jul 16 20:33:41.738290 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:41.738263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:42.493135 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:42.493092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:42.493569 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:42.493092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:42.493569 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:42.493247    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:42.493569 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:42.493269    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:42.729004 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:42.728973    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:33:42.729528 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:42.729475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" event={"ID":"a839f97a-d4f3-4093-8c47-1ab868fce6f9","Type":"ContainerStarted","Data":"261139a147aab48e2820c51d8d423f9aba88ef827b8ff247203b7d8a88971ed4"}
Jul 16 20:33:42.729906 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:42.729875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:42.730074 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:42.730049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:42.746312 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:42.746224    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:33:42.765190 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:42.765119    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx" podStartSLOduration=9.012621895 podStartE2EDuration="28.76508572s" totalImagesPullingTime="19.752463825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:42.764320775 +0000 UTC m=+29.867599761" watchObservedRunningTime="2026-07-16 20:33:42.76508572 +0000 UTC m=+29.868364706"
Jul 16 20:33:43.493689 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.493647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:43.494181 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:43.493758    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:43.674582 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.674552    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dv7n5"]
Jul 16 20:33:43.674750 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.674656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:43.674821 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:43.674746    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:43.675985 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.675958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z4nkd"]
Jul 16 20:33:43.685053 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.684981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4m8f"]
Jul 16 20:33:43.685164 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.685079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:43.685225 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:43.685171    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:43.733231 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.733199    2580 generic.go:356] "Generic (PLEG): container finished" podID="230c5d01-b3a9-4a57-a642-4ac6905b4b4b" containerID="9261d04ff855049e9c3e29d266ac21191ffd4eb50484dfaa875b1b7cee906d89" exitCode=0
Jul 16 20:33:43.733422 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.733246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerDied","Data":"9261d04ff855049e9c3e29d266ac21191ffd4eb50484dfaa875b1b7cee906d89"}
Jul 16 20:33:43.733422 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:43.733410    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:43.733999 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:43.733835    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:45.492881 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:45.492713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:45.493255 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:45.492713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:45.493255 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:45.492754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:45.493255 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:45.493019    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:45.493255 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:45.493081    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:45.493255 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:45.493182    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:45.740522 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:45.740422    2580 generic.go:356] "Generic (PLEG): container finished" podID="230c5d01-b3a9-4a57-a642-4ac6905b4b4b" containerID="0438fc6bd745f1abb991976384fa685d6154a6e2b772bb21ed35a5d6204ad6d6" exitCode=0
Jul 16 20:33:45.740659 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:45.740499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerDied","Data":"0438fc6bd745f1abb991976384fa685d6154a6e2b772bb21ed35a5d6204ad6d6"}
Jul 16 20:33:47.050962 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:47.050924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:47.051676 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:47.050980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:47.051676 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.051060    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:47.051676 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.051129    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs podName:8266a436-acb6-4c37-82be-9cccbbb1e9bc nodeName:}" failed. No retries permitted until 2026-07-16 20:34:19.051113141 +0000 UTC m=+66.154392108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs") pod "network-metrics-daemon-c4m8f" (UID: "8266a436-acb6-4c37-82be-9cccbbb1e9bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:47.051676 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.051136    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:47.051676 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.051199    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret podName:783a85c6-76a0-49bf-99fc-b1083bdc1bf7 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:19.051181466 +0000 UTC m=+66.154460452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret") pod "global-pull-secret-syncer-dv7n5" (UID: "783a85c6-76a0-49bf-99fc-b1083bdc1bf7") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:47.151492 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:47.151456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:47.151672 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.151645    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:47.151672 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.151665    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:47.151780 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.151676    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf6bx for pod openshift-network-diagnostics/network-check-target-z4nkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:47.151780 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.151734    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx podName:cf97482e-a61d-4ccd-a005-318f050f2238 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:19.151714555 +0000 UTC m=+66.254993548 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf6bx" (UniqueName: "kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx") pod "network-check-target-z4nkd" (UID: "cf97482e-a61d-4ccd-a005-318f050f2238") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:47.493414 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:47.493386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:47.493590 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:47.493387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:47.493590 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.493492    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:47.493590 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.493578    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:47.493695 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:47.493387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:47.493695 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:47.493658    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:49.492486 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:49.492449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:49.492486 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:49.492469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:49.492955 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:49.492449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:49.492955 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:49.492599    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4nkd" podUID="cf97482e-a61d-4ccd-a005-318f050f2238"
Jul 16 20:33:49.492955 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:49.492724    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4m8f" podUID="8266a436-acb6-4c37-82be-9cccbbb1e9bc"
Jul 16 20:33:49.492955 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:49.492805    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dv7n5" podUID="783a85c6-76a0-49bf-99fc-b1083bdc1bf7"
Jul 16 20:33:49.753596 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:49.753567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerStarted","Data":"f1a36045a6b9747716e4ca872f4b6f4fff16feb5a98038889c9e0ac28904798d"}
Jul 16 20:33:50.228754 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.228719    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeReady"
Jul 16 20:33:50.228921 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.228847    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 20:33:50.259722 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.259690    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb765bf46-n2z77"]
Jul 16 20:33:50.261559 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.261543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.264857 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.264832    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 20:33:50.265025 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.264925    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 20:33:50.265112 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.265097    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v7sr7\""
Jul 16 20:33:50.265296 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.265280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 20:33:50.270319 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.270295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 20:33:50.273574 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.273547    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mw784"]
Jul 16 20:33:50.275073 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.275058    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lvz59"]
Jul 16 20:33:50.275200 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.275182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mw784"
Jul 16 20:33:50.276660 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.276629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.277853 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.277800    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb765bf46-n2z77"]
Jul 16 20:33:50.279442 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.279420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 20:33:50.279617 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.279599    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c96n9\""
Jul 16 20:33:50.279898 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.279734    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 20:33:50.279898 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.279802    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 20:33:50.279898 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.279843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q8qkr\""
Jul 16 20:33:50.280094 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.279984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 20:33:50.280094 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.279865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 20:33:50.288082 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.288060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mw784"]
Jul 16 20:33:50.290749 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.290564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lvz59"]
Jul 16 20:33:50.374396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-image-registry-private-configuration\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.374396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-ca-trust-extracted\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.374639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-bound-sa-token\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.374639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406a92af-2a2b-40c8-8026-a456c8de1b42-tmp-dir\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.374639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b97356d-6fe0-4246-bdf7-f533a896daea-cert\") pod \"ingress-canary-mw784\" (UID: \"3b97356d-6fe0-4246-bdf7-f533a896daea\") " pod="openshift-ingress-canary/ingress-canary-mw784"
Jul 16 20:33:50.374639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-installation-pull-secrets\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.374639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.374639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbdz\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-kube-api-access-dpbdz\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.374639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.374857 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n564t\" (UniqueName: \"kubernetes.io/projected/3b97356d-6fe0-4246-bdf7-f533a896daea-kube-api-access-n564t\") pod \"ingress-canary-mw784\" (UID: \"3b97356d-6fe0-4246-bdf7-f533a896daea\") " pod="openshift-ingress-canary/ingress-canary-mw784"
Jul 16 20:33:50.374857 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/406a92af-2a2b-40c8-8026-a456c8de1b42-config-volume\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.374857 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xnc\" (UniqueName: \"kubernetes.io/projected/406a92af-2a2b-40c8-8026-a456c8de1b42-kube-api-access-j8xnc\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.374857 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-certificates\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.374857 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.374751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-trusted-ca\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.475479 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-installation-pull-secrets\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.475569 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.475569 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbdz\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-kube-api-access-dpbdz\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.475643 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.475643 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n564t\" (UniqueName: \"kubernetes.io/projected/3b97356d-6fe0-4246-bdf7-f533a896daea-kube-api-access-n564t\") pod \"ingress-canary-mw784\" (UID: \"3b97356d-6fe0-4246-bdf7-f533a896daea\") " pod="openshift-ingress-canary/ingress-canary-mw784"
Jul 16 20:33:50.475752 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/406a92af-2a2b-40c8-8026-a456c8de1b42-config-volume\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.475752 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.475678    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:50.475752 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xnc\" (UniqueName: \"kubernetes.io/projected/406a92af-2a2b-40c8-8026-a456c8de1b42-kube-api-access-j8xnc\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.475752 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.475694    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb765bf46-n2z77: secret "image-registry-tls" not found
Jul 16 20:33:50.475752 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.475705    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:50.475752 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-certificates\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.475752 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.475752    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls podName:6db60eec-50b2-4e2c-aec5-3bbc4e981e62 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:50.975730925 +0000 UTC m=+38.079009902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls") pod "image-registry-fb765bf46-n2z77" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62") : secret "image-registry-tls" not found
Jul 16 20:33:50.476060 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.475902    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls podName:406a92af-2a2b-40c8-8026-a456c8de1b42 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:50.975888772 +0000 UTC m=+38.079167736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls") pod "dns-default-lvz59" (UID: "406a92af-2a2b-40c8-8026-a456c8de1b42") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:50.476060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-trusted-ca\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.476060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-image-registry-private-configuration\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.476060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.475989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-ca-trust-extracted\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.476060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.476012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-bound-sa-token\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.476060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.476035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406a92af-2a2b-40c8-8026-a456c8de1b42-tmp-dir\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.476363 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.476065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b97356d-6fe0-4246-bdf7-f533a896daea-cert\") pod \"ingress-canary-mw784\" (UID: \"3b97356d-6fe0-4246-bdf7-f533a896daea\") " pod="openshift-ingress-canary/ingress-canary-mw784"
Jul 16 20:33:50.476363 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.476238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/406a92af-2a2b-40c8-8026-a456c8de1b42-config-volume\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.476363 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.476309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-certificates\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.476503 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.476436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-ca-trust-extracted\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.476699 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.476678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406a92af-2a2b-40c8-8026-a456c8de1b42-tmp-dir\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.477013 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.476987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-trusted-ca\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.479897 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.479844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-image-registry-private-configuration\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.479897 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.479868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b97356d-6fe0-4246-bdf7-f533a896daea-cert\") pod \"ingress-canary-mw784\" (UID: \"3b97356d-6fe0-4246-bdf7-f533a896daea\") " pod="openshift-ingress-canary/ingress-canary-mw784"
Jul 16 20:33:50.479897 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.479864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-installation-pull-secrets\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.484714 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.484668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-bound-sa-token\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.485129 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.485105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbdz\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-kube-api-access-dpbdz\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.485950 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.485930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xnc\" (UniqueName: \"kubernetes.io/projected/406a92af-2a2b-40c8-8026-a456c8de1b42-kube-api-access-j8xnc\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.486505 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.486482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n564t\" (UniqueName: \"kubernetes.io/projected/3b97356d-6fe0-4246-bdf7-f533a896daea-kube-api-access-n564t\") pod \"ingress-canary-mw784\" (UID: \"3b97356d-6fe0-4246-bdf7-f533a896daea\") " pod="openshift-ingress-canary/ingress-canary-mw784"
Jul 16 20:33:50.584905 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.584868    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mw784"
Jul 16 20:33:50.716674 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.715973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mw784"]
Jul 16 20:33:50.719175 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:50.719147    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97356d_6fe0_4246_bdf7_f533a896daea.slice/crio-ce2113ab3f27e8ed11d47f8250a72f9c7d9d99a5bb7aec8cc8f0429ff1951bfb WatchSource:0}: Error finding container ce2113ab3f27e8ed11d47f8250a72f9c7d9d99a5bb7aec8cc8f0429ff1951bfb: Status 404 returned error can't find the container with id ce2113ab3f27e8ed11d47f8250a72f9c7d9d99a5bb7aec8cc8f0429ff1951bfb
Jul 16 20:33:50.757385 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.757265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mw784" event={"ID":"3b97356d-6fe0-4246-bdf7-f533a896daea","Type":"ContainerStarted","Data":"ce2113ab3f27e8ed11d47f8250a72f9c7d9d99a5bb7aec8cc8f0429ff1951bfb"}
Jul 16 20:33:50.759800 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.759777    2580 generic.go:356] "Generic (PLEG): container finished" podID="230c5d01-b3a9-4a57-a642-4ac6905b4b4b" containerID="f1a36045a6b9747716e4ca872f4b6f4fff16feb5a98038889c9e0ac28904798d" exitCode=0
Jul 16 20:33:50.759916 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.759815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerDied","Data":"f1a36045a6b9747716e4ca872f4b6f4fff16feb5a98038889c9e0ac28904798d"}
Jul 16 20:33:50.980119 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.980084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:50.980119 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:50.980122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:50.980334 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.980249    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:50.980334 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.980279    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb765bf46-n2z77: secret "image-registry-tls" not found
Jul 16 20:33:50.980334 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.980253    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:50.980334 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.980333    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls podName:406a92af-2a2b-40c8-8026-a456c8de1b42 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:51.98031963 +0000 UTC m=+39.083598596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls") pod "dns-default-lvz59" (UID: "406a92af-2a2b-40c8-8026-a456c8de1b42") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:50.980460 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:50.980348    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls podName:6db60eec-50b2-4e2c-aec5-3bbc4e981e62 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:51.980341744 +0000 UTC m=+39.083620707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls") pod "image-registry-fb765bf46-n2z77" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62") : secret "image-registry-tls" not found
Jul 16 20:33:51.495998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.495971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:33:51.495998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.495984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:33:51.496256 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.496200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:33:51.499205 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.499182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 20:33:51.499205 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.499198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 20:33:51.499378 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.499233    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 20:33:51.499378 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.499202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ss679\""
Jul 16 20:33:51.499378 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.499189    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xxdx\""
Jul 16 20:33:51.499378 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.499322    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 20:33:51.766748 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.766668    2580 generic.go:356] "Generic (PLEG): container finished" podID="230c5d01-b3a9-4a57-a642-4ac6905b4b4b" containerID="027f698b0a4e05348c3de1823f05a796bbfe058dfa3c13821181dfe4857c17c6" exitCode=0
Jul 16 20:33:51.766748 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.766734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerDied","Data":"027f698b0a4e05348c3de1823f05a796bbfe058dfa3c13821181dfe4857c17c6"}
Jul 16 20:33:51.989843 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.989629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:51.990025 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:51.989862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:51.990025 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:51.989804    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:51.990025 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:51.989933    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb765bf46-n2z77: secret "image-registry-tls" not found
Jul 16 20:33:51.990025 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:51.990000    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls podName:6db60eec-50b2-4e2c-aec5-3bbc4e981e62 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:53.989983696 +0000 UTC m=+41.093262663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls") pod "image-registry-fb765bf46-n2z77" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62") : secret "image-registry-tls" not found
Jul 16 20:33:51.990025 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:51.990001    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:51.990291 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:51.990053    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls podName:406a92af-2a2b-40c8-8026-a456c8de1b42 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:53.990037074 +0000 UTC m=+41.093316051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls") pod "dns-default-lvz59" (UID: "406a92af-2a2b-40c8-8026-a456c8de1b42") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:52.772076 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:52.772042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" event={"ID":"230c5d01-b3a9-4a57-a642-4ac6905b4b4b","Type":"ContainerStarted","Data":"2f04a0041a8f4664a035d72372edbca548ec4da100bd84b4893ddf3325894c9e"}
Jul 16 20:33:52.794570 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:52.794505    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sr7vx" podStartSLOduration=8.881687421 podStartE2EDuration="38.794488154s" totalImagesPullingTime="29.912800733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:52.793079338 +0000 UTC m=+39.896358337" watchObservedRunningTime="2026-07-16 20:33:52.794488154 +0000 UTC m=+39.897767139"
Jul 16 20:33:53.775191 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:53.775152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mw784" event={"ID":"3b97356d-6fe0-4246-bdf7-f533a896daea","Type":"ContainerStarted","Data":"c86ab908280c07ccfac4bd9b934dcf11e802bb13534e8b0ca4feedae947d59c9"}
Jul 16 20:33:53.790895 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:53.790850    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mw784" podStartSLOduration=1.358731316 podStartE2EDuration="3.790835659s" totalImagesPullingTime="2.432104343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:53.789944812 +0000 UTC m=+40.893223797" watchObservedRunningTime="2026-07-16 20:33:53.790835659 +0000 UTC m=+40.894114643"
Jul 16 20:33:54.006017 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:54.005974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:54.006017 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:54.006021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:54.006252 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:54.006137    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:54.006252 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:54.006162    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb765bf46-n2z77: secret "image-registry-tls" not found
Jul 16 20:33:54.006252 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:54.006139    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:54.006252 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:54.006218    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls podName:6db60eec-50b2-4e2c-aec5-3bbc4e981e62 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:58.006201208 +0000 UTC m=+45.109480173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls") pod "image-registry-fb765bf46-n2z77" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62") : secret "image-registry-tls" not found
Jul 16 20:33:54.006252 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:54.006230    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls podName:406a92af-2a2b-40c8-8026-a456c8de1b42 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:58.006224654 +0000 UTC m=+45.109503617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls") pod "dns-default-lvz59" (UID: "406a92af-2a2b-40c8-8026-a456c8de1b42") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:57.384271 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.384236    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5htcq"]
Jul 16 20:33:57.419830 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.419789    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5htcq"]
Jul 16 20:33:57.419984 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.419922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.424465 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.424445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 20:33:57.424996 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.424981    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 20:33:57.425524 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.425500    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 20:33:57.426825 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.426809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 20:33:57.433433 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.433411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ljszm\""
Jul 16 20:33:57.532814 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.532779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.532983 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.532834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-data-volume\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.532983 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.532963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4s8\" (UniqueName: \"kubernetes.io/projected/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-api-access-7t4s8\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.533085 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.533002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.533085 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.533029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-crio-socket\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.634142 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.634101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-data-volume\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.634351 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.634197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4s8\" (UniqueName: \"kubernetes.io/projected/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-api-access-7t4s8\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.634351 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.634231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.634351 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.634266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-crio-socket\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.634351 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.634307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.634589 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:57.634355    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:33:57.634589 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:57.634417    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:57.634589 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:57.634448    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:58.134429922 +0000 UTC m=+45.237708903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : configmap "kube-rbac-proxy" not found
Jul 16 20:33:57.634589 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:57.634477    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:58.134461456 +0000 UTC m=+45.237740421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:57.634589 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.634486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-data-volume\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.634589 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.634555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-crio-socket\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:57.645585 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:57.645557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4s8\" (UniqueName: \"kubernetes.io/projected/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-api-access-7t4s8\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:58.038230 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.038140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:33:58.038230 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.038182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:33:58.038390 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.038298    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:58.038390 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.038366    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls podName:406a92af-2a2b-40c8-8026-a456c8de1b42 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:06.038352422 +0000 UTC m=+53.141631388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls") pod "dns-default-lvz59" (UID: "406a92af-2a2b-40c8-8026-a456c8de1b42") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:58.038390 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.038296    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:58.038390 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.038386    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb765bf46-n2z77: secret "image-registry-tls" not found
Jul 16 20:33:58.038542 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.038419    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls podName:6db60eec-50b2-4e2c-aec5-3bbc4e981e62 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:06.038409305 +0000 UTC m=+53.141688283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls") pod "image-registry-fb765bf46-n2z77" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62") : secret "image-registry-tls" not found
Jul 16 20:33:58.139451 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.139416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:58.139614 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.139550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:58.139614 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.139588    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:58.139698 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.139645    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:33:58.139698 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.139687    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:59.139669018 +0000 UTC m=+46.242947987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:58.139760 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:58.139702    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:59.139696498 +0000 UTC m=+46.242975461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : configmap "kube-rbac-proxy" not found
Jul 16 20:33:58.301862 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.301785    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b"]
Jul 16 20:33:58.321313 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.321289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b"]
Jul 16 20:33:58.321481 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.321382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b"
Jul 16 20:33:58.324112 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.324095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dbt47\""
Jul 16 20:33:58.324428 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.324413    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 16 20:33:58.324557 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.324416    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 16 20:33:58.341410 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.341383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snjt\" (UniqueName: \"kubernetes.io/projected/bea279e7-635d-483f-9809-301988f923a7-kube-api-access-8snjt\") pod \"migrator-7cc9c7ffd7-7p49b\" (UID: \"bea279e7-635d-483f-9809-301988f923a7\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b"
Jul 16 20:33:58.442813 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.442775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snjt\" (UniqueName: \"kubernetes.io/projected/bea279e7-635d-483f-9809-301988f923a7-kube-api-access-8snjt\") pod \"migrator-7cc9c7ffd7-7p49b\" (UID: \"bea279e7-635d-483f-9809-301988f923a7\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b"
Jul 16 20:33:58.460468 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.460443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snjt\" (UniqueName: \"kubernetes.io/projected/bea279e7-635d-483f-9809-301988f923a7-kube-api-access-8snjt\") pod \"migrator-7cc9c7ffd7-7p49b\" (UID: \"bea279e7-635d-483f-9809-301988f923a7\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b"
Jul 16 20:33:58.630918 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.630840    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b"
Jul 16 20:33:58.749947 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.749916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b"]
Jul 16 20:33:58.752965 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:33:58.752923    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea279e7_635d_483f_9809_301988f923a7.slice/crio-88387d126bbe3129655976b6f0acc843571db51c0b54b33886c50adbebbd5392 WatchSource:0}: Error finding container 88387d126bbe3129655976b6f0acc843571db51c0b54b33886c50adbebbd5392: Status 404 returned error can't find the container with id 88387d126bbe3129655976b6f0acc843571db51c0b54b33886c50adbebbd5392
Jul 16 20:33:58.788123 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:58.788093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b" event={"ID":"bea279e7-635d-483f-9809-301988f923a7","Type":"ContainerStarted","Data":"88387d126bbe3129655976b6f0acc843571db51c0b54b33886c50adbebbd5392"}
Jul 16 20:33:59.143630 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:59.143603    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x2tl9_93410d88-85b1-4b03-b66b-d6de95740d86/dns-node-resolver/0.log"
Jul 16 20:33:59.148937 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:59.148917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:59.148999 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:59.148965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:33:59.149062 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:59.149046    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:33:59.149096 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:59.149065    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:59.149184 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:59.149105    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:01.149093013 +0000 UTC m=+48.252371975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:59.149184 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:33:59.149122    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:01.149112985 +0000 UTC m=+48.252391948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : configmap "kube-rbac-proxy" not found
Jul 16 20:33:59.944099 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:33:59.944074    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rnthw_24a3af80-d268-4bb1-8a80-b6a69d5d6db2/node-ca/0.log"
Jul 16 20:34:00.744638 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:00.744605    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mw784_3b97356d-6fe0-4246-bdf7-f533a896daea/serve-healthcheck-canary/0.log"
Jul 16 20:34:01.163261 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:01.163229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:34:01.163676 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:01.163332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:34:01.163676 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:01.163415    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:34:01.163676 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:01.163470    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:05.163455403 +0000 UTC m=+52.266734366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : configmap "kube-rbac-proxy" not found
Jul 16 20:34:01.163676 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:01.163417    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:34:01.163676 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:01.163563    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:05.163551324 +0000 UTC m=+52.266830301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:34:02.799813 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:02.799776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b" event={"ID":"bea279e7-635d-483f-9809-301988f923a7","Type":"ContainerStarted","Data":"5111fed98a213e336999df06a6bc275a634292b17a585be247f3be42e805546c"}
Jul 16 20:34:02.800217 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:02.799857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b" event={"ID":"bea279e7-635d-483f-9809-301988f923a7","Type":"ContainerStarted","Data":"b5d5873e47b786b5987693077154e171e305426200658e83465db597e4bf686b"}
Jul 16 20:34:02.817172 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:02.817120    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7p49b" podStartSLOduration=1.211709306 podStartE2EDuration="4.817104022s" totalImagesPullingTime="3.605394716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:02.816568396 +0000 UTC m=+49.919847378" watchObservedRunningTime="2026-07-16 20:34:02.817104022 +0000 UTC m=+49.920383007"
Jul 16 20:34:05.195763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:05.195726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:34:05.196167 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:05.195782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:34:05.196167 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:05.195877    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:34:05.196167 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:05.195929    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls podName:dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:13.195915474 +0000 UTC m=+60.299194437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5htcq" (UID: "dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:34:05.196289 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:05.196262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:34:06.101906 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:06.101872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:34:06.101906 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:06.101909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:34:06.102190 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:06.102006    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:34:06.102190 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:06.102016    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:34:06.102190 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:06.102035    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb765bf46-n2z77: secret "image-registry-tls" not found
Jul 16 20:34:06.102190 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:06.102057    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls podName:406a92af-2a2b-40c8-8026-a456c8de1b42 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:22.102043113 +0000 UTC m=+69.205322076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls") pod "dns-default-lvz59" (UID: "406a92af-2a2b-40c8-8026-a456c8de1b42") : secret "dns-default-metrics-tls" not found
Jul 16 20:34:06.102190 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:06.102076    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls podName:6db60eec-50b2-4e2c-aec5-3bbc4e981e62 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:22.102062912 +0000 UTC m=+69.205341875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls") pod "image-registry-fb765bf46-n2z77" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62") : secret "image-registry-tls" not found
Jul 16 20:34:13.254402 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:13.254369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:34:13.256574 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:13.256557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5htcq\" (UID: \"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3\") " pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:34:13.331375 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:13.331343    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ljszm\""
Jul 16 20:34:13.339549 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:13.339508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5htcq"
Jul 16 20:34:13.461968 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:13.461938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5htcq"]
Jul 16 20:34:13.465069 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:13.465030    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd44ffcb_1f3b_4b0d_a1ac_e7d26171e8a3.slice/crio-1c33185bcf3967acf5d65c2f13eed8bdc0e5e7c703ac346e4a241fb38350b2a0 WatchSource:0}: Error finding container 1c33185bcf3967acf5d65c2f13eed8bdc0e5e7c703ac346e4a241fb38350b2a0: Status 404 returned error can't find the container with id 1c33185bcf3967acf5d65c2f13eed8bdc0e5e7c703ac346e4a241fb38350b2a0
Jul 16 20:34:13.825285 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:13.825250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5htcq" event={"ID":"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3","Type":"ContainerStarted","Data":"1c33185bcf3967acf5d65c2f13eed8bdc0e5e7c703ac346e4a241fb38350b2a0"}
Jul 16 20:34:14.754537 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:14.754489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qszx"
Jul 16 20:34:19.102505 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.102462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:34:19.102996 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.102533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:34:19.105425 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.105405    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 20:34:19.106232 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.106218    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 20:34:19.115297 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.115273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/783a85c6-76a0-49bf-99fc-b1083bdc1bf7-original-pull-secret\") pod \"global-pull-secret-syncer-dv7n5\" (UID: \"783a85c6-76a0-49bf-99fc-b1083bdc1bf7\") " pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:34:19.115375 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.115332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8266a436-acb6-4c37-82be-9cccbbb1e9bc-metrics-certs\") pod \"network-metrics-daemon-c4m8f\" (UID: \"8266a436-acb6-4c37-82be-9cccbbb1e9bc\") " pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:34:19.121710 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.121691    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ss679\""
Jul 16 20:34:19.129729 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.129715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4m8f"
Jul 16 20:34:19.203846 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.203811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:34:19.208501 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.208475    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 20:34:19.217252 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.217225    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 20:34:19.227472 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.227440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf6bx\" (UniqueName: \"kubernetes.io/projected/cf97482e-a61d-4ccd-a005-318f050f2238-kube-api-access-lf6bx\") pod \"network-check-target-z4nkd\" (UID: \"cf97482e-a61d-4ccd-a005-318f050f2238\") " pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:34:19.410011 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.409984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xxdx\""
Jul 16 20:34:19.413393 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.413373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dv7n5"
Jul 16 20:34:19.418371 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.418350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:34:19.943806 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:19.943753    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4m8f"]
Jul 16 20:34:19.947644 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:19.947615    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8266a436_acb6_4c37_82be_9cccbbb1e9bc.slice/crio-78bf6fa863f2b2646b146b1208a8fe7c45bb8fbcbce40abe0f5b405d2c8bb297 WatchSource:0}: Error finding container 78bf6fa863f2b2646b146b1208a8fe7c45bb8fbcbce40abe0f5b405d2c8bb297: Status 404 returned error can't find the container with id 78bf6fa863f2b2646b146b1208a8fe7c45bb8fbcbce40abe0f5b405d2c8bb297
Jul 16 20:34:20.165353 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:20.165326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dv7n5"]
Jul 16 20:34:20.168615 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:20.168508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z4nkd"]
Jul 16 20:34:20.170816 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:20.170790    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783a85c6_76a0_49bf_99fc_b1083bdc1bf7.slice/crio-6a5762e2d427ace663790bcf254dd74540a5d34b4e7611c0bb7c5698eb389510 WatchSource:0}: Error finding container 6a5762e2d427ace663790bcf254dd74540a5d34b4e7611c0bb7c5698eb389510: Status 404 returned error can't find the container with id 6a5762e2d427ace663790bcf254dd74540a5d34b4e7611c0bb7c5698eb389510
Jul 16 20:34:20.174232 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:20.174206    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf97482e_a61d_4ccd_a005_318f050f2238.slice/crio-7249139a2290739e1f19c7c31a0ebb5f2ba6c6d5dec22002ed997dcd9e4836cd WatchSource:0}: Error finding container 7249139a2290739e1f19c7c31a0ebb5f2ba6c6d5dec22002ed997dcd9e4836cd: Status 404 returned error can't find the container with id 7249139a2290739e1f19c7c31a0ebb5f2ba6c6d5dec22002ed997dcd9e4836cd
Jul 16 20:34:20.847880 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:20.847837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5htcq" event={"ID":"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3","Type":"ContainerStarted","Data":"4994a6a060b31a77ee5043c8b4a5aff25e111820527a121429a5c0c1d470632e"}
Jul 16 20:34:20.849416 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:20.849389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z4nkd" event={"ID":"cf97482e-a61d-4ccd-a005-318f050f2238","Type":"ContainerStarted","Data":"7249139a2290739e1f19c7c31a0ebb5f2ba6c6d5dec22002ed997dcd9e4836cd"}
Jul 16 20:34:20.850669 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:20.850624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dv7n5" event={"ID":"783a85c6-76a0-49bf-99fc-b1083bdc1bf7","Type":"ContainerStarted","Data":"6a5762e2d427ace663790bcf254dd74540a5d34b4e7611c0bb7c5698eb389510"}
Jul 16 20:34:20.852131 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:20.852080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4m8f" event={"ID":"8266a436-acb6-4c37-82be-9cccbbb1e9bc","Type":"ContainerStarted","Data":"78bf6fa863f2b2646b146b1208a8fe7c45bb8fbcbce40abe0f5b405d2c8bb297"}
Jul 16 20:34:22.125969 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:22.125928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:34:22.126426 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:22.125986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:34:22.130037 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:22.129983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"image-registry-fb765bf46-n2z77\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") " pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:34:22.131782 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:22.131757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/406a92af-2a2b-40c8-8026-a456c8de1b42-metrics-tls\") pod \"dns-default-lvz59\" (UID: \"406a92af-2a2b-40c8-8026-a456c8de1b42\") " pod="openshift-dns/dns-default-lvz59"
Jul 16 20:34:22.373689 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:22.373655    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v7sr7\""
Jul 16 20:34:22.381949 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:22.381875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:34:22.393679 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:22.393415    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q8qkr\""
Jul 16 20:34:22.401767 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:22.401580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lvz59"
Jul 16 20:34:23.920435 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.920405    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"]
Jul 16 20:34:23.949940 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.949906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"]
Jul 16 20:34:23.950096 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.949951    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"]
Jul 16 20:34:23.950096 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.950050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:23.954260 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.954062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 16 20:34:23.954260 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.954136    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 20:34:23.954260 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.954168    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 20:34:23.954260 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.954190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 20:34:23.963584 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.963561    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb765bf46-n2z77"]
Jul 16 20:34:23.963695 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.963606    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"]
Jul 16 20:34:23.963695 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.963673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"
Jul 16 20:34:23.966477 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.966453    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ps2ch\""
Jul 16 20:34:23.966930 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:23.966905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 16 20:34:24.026348 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.026314    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-h6r9r"]
Jul 16 20:34:24.040638 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.040580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f2319d8-0d37-4cd7-9008-4623fc16e16c-tmp\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.040638 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.040618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1f2319d8-0d37-4cd7-9008-4623fc16e16c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.040836 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.040737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62b5c\" (UniqueName: \"kubernetes.io/projected/1f2319d8-0d37-4cd7-9008-4623fc16e16c-kube-api-access-62b5c\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.040836 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.040768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbd0b16a-739f-456e-bd9a-66807938903e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg\" (UID: \"cbd0b16a-739f-456e-bd9a-66807938903e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"
Jul 16 20:34:24.040836 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.040792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4l8z\" (UniqueName: \"kubernetes.io/projected/cbd0b16a-739f-456e-bd9a-66807938903e-kube-api-access-d4l8z\") pod \"managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg\" (UID: \"cbd0b16a-739f-456e-bd9a-66807938903e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"
Jul 16 20:34:24.046315 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.046143    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-h6r9r"]
Jul 16 20:34:24.046315 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.046253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-h6r9r"
Jul 16 20:34:24.049835 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.049812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7hvxw\""
Jul 16 20:34:24.049952 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.049889    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 20:34:24.049952 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.049898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 20:34:24.141452 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.141415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62b5c\" (UniqueName: \"kubernetes.io/projected/1f2319d8-0d37-4cd7-9008-4623fc16e16c-kube-api-access-62b5c\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.141665 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.141470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbd0b16a-739f-456e-bd9a-66807938903e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg\" (UID: \"cbd0b16a-739f-456e-bd9a-66807938903e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"
Jul 16 20:34:24.141665 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.141501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4l8z\" (UniqueName: \"kubernetes.io/projected/cbd0b16a-739f-456e-bd9a-66807938903e-kube-api-access-d4l8z\") pod \"managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg\" (UID: \"cbd0b16a-739f-456e-bd9a-66807938903e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"
Jul 16 20:34:24.141665 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.141549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcdnr\" (UniqueName: \"kubernetes.io/projected/30ab91e5-66ba-426d-bd20-e4767ef0202c-kube-api-access-rcdnr\") pod \"downloads-5bc67dcd46-h6r9r\" (UID: \"30ab91e5-66ba-426d-bd20-e4767ef0202c\") " pod="openshift-console/downloads-5bc67dcd46-h6r9r"
Jul 16 20:34:24.141665 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.141592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f2319d8-0d37-4cd7-9008-4623fc16e16c-tmp\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.141665 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.141616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1f2319d8-0d37-4cd7-9008-4623fc16e16c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.142066 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.142017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f2319d8-0d37-4cd7-9008-4623fc16e16c-tmp\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.144540 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.144501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbd0b16a-739f-456e-bd9a-66807938903e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg\" (UID: \"cbd0b16a-739f-456e-bd9a-66807938903e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"
Jul 16 20:34:24.144640 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.144615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1f2319d8-0d37-4cd7-9008-4623fc16e16c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.153588 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.153560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4l8z\" (UniqueName: \"kubernetes.io/projected/cbd0b16a-739f-456e-bd9a-66807938903e-kube-api-access-d4l8z\") pod \"managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg\" (UID: \"cbd0b16a-739f-456e-bd9a-66807938903e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"
Jul 16 20:34:24.153997 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.153977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62b5c\" (UniqueName: \"kubernetes.io/projected/1f2319d8-0d37-4cd7-9008-4623fc16e16c-kube-api-access-62b5c\") pod \"klusterlet-addon-workmgr-6cc7c6848d-4l8cd\" (UID: \"1f2319d8-0d37-4cd7-9008-4623fc16e16c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.242862 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.242779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcdnr\" (UniqueName: \"kubernetes.io/projected/30ab91e5-66ba-426d-bd20-e4767ef0202c-kube-api-access-rcdnr\") pod \"downloads-5bc67dcd46-h6r9r\" (UID: \"30ab91e5-66ba-426d-bd20-e4767ef0202c\") " pod="openshift-console/downloads-5bc67dcd46-h6r9r"
Jul 16 20:34:24.253047 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.253016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcdnr\" (UniqueName: \"kubernetes.io/projected/30ab91e5-66ba-426d-bd20-e4767ef0202c-kube-api-access-rcdnr\") pod \"downloads-5bc67dcd46-h6r9r\" (UID: \"30ab91e5-66ba-426d-bd20-e4767ef0202c\") " pod="openshift-console/downloads-5bc67dcd46-h6r9r"
Jul 16 20:34:24.262105 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.262077    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:24.290848 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.290809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"
Jul 16 20:34:24.357108 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:24.357075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-h6r9r"
Jul 16 20:34:26.707315 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.707167    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lvz59"]
Jul 16 20:34:26.722044 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.721975    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb765bf46-n2z77"]
Jul 16 20:34:26.724891 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:26.724853    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db60eec_50b2_4e2c_aec5_3bbc4e981e62.slice/crio-603774c47f3bb8e4f88a2b3d59bdd827c611cc0e63f22a1e3557727730ccb00f WatchSource:0}: Error finding container 603774c47f3bb8e4f88a2b3d59bdd827c611cc0e63f22a1e3557727730ccb00f: Status 404 returned error can't find the container with id 603774c47f3bb8e4f88a2b3d59bdd827c611cc0e63f22a1e3557727730ccb00f
Jul 16 20:34:26.740249 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.740225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"]
Jul 16 20:34:26.746029 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:26.745991    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2319d8_0d37_4cd7_9008_4623fc16e16c.slice/crio-b18ba39ebf74426caf46c4493167ea96e9b7a7d56882065cb73cadc620006ab4 WatchSource:0}: Error finding container b18ba39ebf74426caf46c4493167ea96e9b7a7d56882065cb73cadc620006ab4: Status 404 returned error can't find the container with id b18ba39ebf74426caf46c4493167ea96e9b7a7d56882065cb73cadc620006ab4
Jul 16 20:34:26.757966 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.757937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-h6r9r"]
Jul 16 20:34:26.774891 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.774868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg"]
Jul 16 20:34:26.781194 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:26.781167    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbd0b16a_739f_456e_bd9a_66807938903e.slice/crio-ce705206d6d2b6a97dc2b79a52c68d9adaeab731aa049f52509899428602a317 WatchSource:0}: Error finding container ce705206d6d2b6a97dc2b79a52c68d9adaeab731aa049f52509899428602a317: Status 404 returned error can't find the container with id ce705206d6d2b6a97dc2b79a52c68d9adaeab731aa049f52509899428602a317
Jul 16 20:34:26.874795 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.874763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z4nkd" event={"ID":"cf97482e-a61d-4ccd-a005-318f050f2238","Type":"ContainerStarted","Data":"d6760e87e4f59f22ae276154db81417e9e05e822daa9868a34fdb2059d86661b"}
Jul 16 20:34:26.874973 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.874872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:34:26.876609 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.876575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dv7n5" event={"ID":"783a85c6-76a0-49bf-99fc-b1083bdc1bf7","Type":"ContainerStarted","Data":"630f9fc6a79011e8ab7f8fd1e439f1ed463e924763809f7e83c86fe1e5182792"}
Jul 16 20:34:26.881659 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.881630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4m8f" event={"ID":"8266a436-acb6-4c37-82be-9cccbbb1e9bc","Type":"ContainerStarted","Data":"d74969668ed0d21d1ad0f811fc116dadd5529d71fbb8a6470e5956efdb786f1d"}
Jul 16 20:34:26.881784 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.881677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4m8f" event={"ID":"8266a436-acb6-4c37-82be-9cccbbb1e9bc","Type":"ContainerStarted","Data":"8180feed28796c690424334c84b80456e266859ca6277a935a93b18e88ece86b"}
Jul 16 20:34:26.882780 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.882755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-h6r9r" event={"ID":"30ab91e5-66ba-426d-bd20-e4767ef0202c","Type":"ContainerStarted","Data":"bde59ccffb2826d8180a5032603f9ba568f9c27bae8c51f1d8a4b57922101251"}
Jul 16 20:34:26.883977 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.883949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd" event={"ID":"1f2319d8-0d37-4cd7-9008-4623fc16e16c","Type":"ContainerStarted","Data":"b18ba39ebf74426caf46c4493167ea96e9b7a7d56882065cb73cadc620006ab4"}
Jul 16 20:34:26.884988 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.884968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg" event={"ID":"cbd0b16a-739f-456e-bd9a-66807938903e","Type":"ContainerStarted","Data":"ce705206d6d2b6a97dc2b79a52c68d9adaeab731aa049f52509899428602a317"}
Jul 16 20:34:26.886474 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.886453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb765bf46-n2z77" event={"ID":"6db60eec-50b2-4e2c-aec5-3bbc4e981e62","Type":"ContainerStarted","Data":"71f8d66c45e356bbbedbbb14c9d8d57548d0196f1127870388240496da71ff97"}
Jul 16 20:34:26.886583 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.886490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb765bf46-n2z77" event={"ID":"6db60eec-50b2-4e2c-aec5-3bbc4e981e62","Type":"ContainerStarted","Data":"603774c47f3bb8e4f88a2b3d59bdd827c611cc0e63f22a1e3557727730ccb00f"}
Jul 16 20:34:26.886583 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.886533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:34:26.888486 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.888465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5htcq" event={"ID":"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3","Type":"ContainerStarted","Data":"ef8f19d3c4651be9a9e563646108932c11aa6e6fd40b079bf232546f8a65b8f9"}
Jul 16 20:34:26.889762 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.889739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvz59" event={"ID":"406a92af-2a2b-40c8-8026-a456c8de1b42","Type":"ContainerStarted","Data":"1717e5bcc6a84e8b6de17b0dcd75118d3b5a130db28a82741829e0d49bdb919c"}
Jul 16 20:34:26.891252 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.891212    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z4nkd" podStartSLOduration=66.581027491 podStartE2EDuration="1m12.891200886s" totalImagesPullingTime="6.310173395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:26.889613991 +0000 UTC m=+73.992892977" watchObservedRunningTime="2026-07-16 20:34:26.891200886 +0000 UTC m=+73.994479870"
Jul 16 20:34:26.911725 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.911630    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c4m8f" podStartSLOduration=66.376587311 podStartE2EDuration="1m12.911610614s" totalImagesPullingTime="6.535023303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:26.910910621 +0000 UTC m=+74.014189630" watchObservedRunningTime="2026-07-16 20:34:26.911610614 +0000 UTC m=+74.014889604"
Jul 16 20:34:26.934303 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.934255    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fb765bf46-n2z77" podStartSLOduration=66.934240343 podStartE2EDuration="1m6.934240343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:26.933446164 +0000 UTC m=+74.036725149" watchObservedRunningTime="2026-07-16 20:34:26.934240343 +0000 UTC m=+74.037519328"
Jul 16 20:34:26.948831 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:26.948783    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dv7n5" podStartSLOduration=66.609122945 podStartE2EDuration="1m12.948769076s" totalImagesPullingTime="6.339646131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:26.948077121 +0000 UTC m=+74.051356117" watchObservedRunningTime="2026-07-16 20:34:26.948769076 +0000 UTC m=+74.052048039"
Jul 16 20:34:28.264551 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.264387    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"]
Jul 16 20:34:28.267976 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.267306    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"
Jul 16 20:34:28.270988 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.270685    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hsltr\""
Jul 16 20:34:28.271092 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.271008    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 16 20:34:28.279555 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.278317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"]
Jul 16 20:34:28.377807 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.377764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5736e36f-2309-45b3-8051-9796822e2c5d-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-tgrlf\" (UID: \"5736e36f-2309-45b3-8051-9796822e2c5d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"
Jul 16 20:34:28.478702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.478661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5736e36f-2309-45b3-8051-9796822e2c5d-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-tgrlf\" (UID: \"5736e36f-2309-45b3-8051-9796822e2c5d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"
Jul 16 20:34:28.478961 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:28.478814    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 16 20:34:28.478961 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:28.478882    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5736e36f-2309-45b3-8051-9796822e2c5d-tls-certificates podName:5736e36f-2309-45b3-8051-9796822e2c5d nodeName:}" failed. No retries permitted until 2026-07-16 20:34:28.978860686 +0000 UTC m=+76.082139653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/5736e36f-2309-45b3-8051-9796822e2c5d-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-tgrlf" (UID: "5736e36f-2309-45b3-8051-9796822e2c5d") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 16 20:34:28.983042 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.983002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5736e36f-2309-45b3-8051-9796822e2c5d-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-tgrlf\" (UID: \"5736e36f-2309-45b3-8051-9796822e2c5d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"
Jul 16 20:34:28.989622 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:28.989559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5736e36f-2309-45b3-8051-9796822e2c5d-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-tgrlf\" (UID: \"5736e36f-2309-45b3-8051-9796822e2c5d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"
Jul 16 20:34:29.188329 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:29.188098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"
Jul 16 20:34:33.760961 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:33.760854    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"]
Jul 16 20:34:34.176616 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:34.176582    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5736e36f_2309_45b3_8051_9796822e2c5d.slice/crio-4b61d5482cbd1a5d153afb59ebb5104405f4048d712615bce6294a18891ece6b WatchSource:0}: Error finding container 4b61d5482cbd1a5d153afb59ebb5104405f4048d712615bce6294a18891ece6b: Status 404 returned error can't find the container with id 4b61d5482cbd1a5d153afb59ebb5104405f4048d712615bce6294a18891ece6b
Jul 16 20:34:34.931183 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.931139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd" event={"ID":"1f2319d8-0d37-4cd7-9008-4623fc16e16c","Type":"ContainerStarted","Data":"3facafd55bfa04c1dbd703f07ad7578c5ba4d6308c5dcbb820635b9c05839ccf"}
Jul 16 20:34:34.931806 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.931757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:34.933696 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.933677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd"
Jul 16 20:34:34.934263 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.934203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg" event={"ID":"cbd0b16a-739f-456e-bd9a-66807938903e","Type":"ContainerStarted","Data":"8b95a22a85070d0864f1a98bb7aac2da1c85f5be7d034e9a42c7d3565ccb1bca"}
Jul 16 20:34:34.937114 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.937087    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5htcq" event={"ID":"dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3","Type":"ContainerStarted","Data":"18627bfbc3298da654fb7193c873c11e726466bfceaed755e098e2b31da6ebd0"}
Jul 16 20:34:34.939072 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.938998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvz59" event={"ID":"406a92af-2a2b-40c8-8026-a456c8de1b42","Type":"ContainerStarted","Data":"f28e1c7ed57c463a3bb9a42da5529a2a59e75f11f51031fcab7eefd461652911"}
Jul 16 20:34:34.939072 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.939042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvz59" event={"ID":"406a92af-2a2b-40c8-8026-a456c8de1b42","Type":"ContainerStarted","Data":"30b3973fc56bdcdc7312603e86bbb67ef778810abddada74fd4bab347d4fadaf"}
Jul 16 20:34:34.939220 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.939192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lvz59"
Jul 16 20:34:34.940417 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.940376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf" event={"ID":"5736e36f-2309-45b3-8051-9796822e2c5d","Type":"ContainerStarted","Data":"4b61d5482cbd1a5d153afb59ebb5104405f4048d712615bce6294a18891ece6b"}
Jul 16 20:34:34.949216 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.949178    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cc7c6848d-4l8cd" podStartSLOduration=4.521291769 podStartE2EDuration="11.949164433s" totalImagesPullingTime="7.427872664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:34.947765691 +0000 UTC m=+82.051044678" watchObservedRunningTime="2026-07-16 20:34:34.949164433 +0000 UTC m=+82.052443418"
Jul 16 20:34:34.982148 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.982094    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lvz59" podStartSLOduration=37.463368392 podStartE2EDuration="44.982079021s" totalImagesPullingTime="7.518710629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:34.966017089 +0000 UTC m=+82.069296075" watchObservedRunningTime="2026-07-16 20:34:34.982079021 +0000 UTC m=+82.085358008"
Jul 16 20:34:34.983274 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:34.983002    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5htcq" podStartSLOduration=17.494751692 podStartE2EDuration="37.982987606s" totalImagesPullingTime="20.488235914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:34.981633784 +0000 UTC m=+82.084912789" watchObservedRunningTime="2026-07-16 20:34:34.982987606 +0000 UTC m=+82.086266591"
Jul 16 20:34:35.013478 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:35.013423    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9c67b6fcd-f8sjg" podStartSLOduration=4.6193866329999995 podStartE2EDuration="12.013402484s" totalImagesPullingTime="7.394015851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:35.012908022 +0000 UTC m=+82.116187008" watchObservedRunningTime="2026-07-16 20:34:35.013402484 +0000 UTC m=+82.116681470"
Jul 16 20:34:35.946206 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:35.946135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf" event={"ID":"5736e36f-2309-45b3-8051-9796822e2c5d","Type":"ContainerStarted","Data":"100d664f76cf29fd3b89dd6d3166fcbdc5a83e7caa185fc0749a85db0e647cff"}
Jul 16 20:34:35.946930 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:35.946880    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"
Jul 16 20:34:35.951751 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:35.951721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf"
Jul 16 20:34:35.961147 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:35.961076    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-tgrlf" podStartSLOduration=6.799841333 podStartE2EDuration="7.961060719s" totalImagesPullingTime="1.161219386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:35.960145008 +0000 UTC m=+83.063424006" watchObservedRunningTime="2026-07-16 20:34:35.961060719 +0000 UTC m=+83.064339682"
Jul 16 20:34:36.318117 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.318035    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"]
Jul 16 20:34:36.321912 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.321885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.324809 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.324778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 20:34:36.324809 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.324801    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 20:34:36.324976 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.324819    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 20:34:36.325138 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.325122    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 20:34:36.325825 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.325803    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 20:34:36.325927 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.325849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gwj8w\""
Jul 16 20:34:36.331341 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.331321    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"]
Jul 16 20:34:36.446560 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.446490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.446560 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.446564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2038113a-1983-4fb3-93f9-ef7d9b40cd07-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.446791 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.446609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.446791 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.446633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xm8t\" (UniqueName: \"kubernetes.io/projected/2038113a-1983-4fb3-93f9-ef7d9b40cd07-kube-api-access-2xm8t\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.547413 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.547373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.547413 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.547411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2038113a-1983-4fb3-93f9-ef7d9b40cd07-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.547667 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.547443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.547667 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.547470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xm8t\" (UniqueName: \"kubernetes.io/projected/2038113a-1983-4fb3-93f9-ef7d9b40cd07-kube-api-access-2xm8t\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.547667 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:36.547572    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 16 20:34:36.547667 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:36.547664    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-tls podName:2038113a-1983-4fb3-93f9-ef7d9b40cd07 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:37.04764099 +0000 UTC m=+84.150919962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-6pm2r" (UID: "2038113a-1983-4fb3-93f9-ef7d9b40cd07") : secret "prometheus-operator-tls" not found
Jul 16 20:34:36.548209 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.548189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2038113a-1983-4fb3-93f9-ef7d9b40cd07-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.550089 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.550052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:36.557187 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:36.557161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xm8t\" (UniqueName: \"kubernetes.io/projected/2038113a-1983-4fb3-93f9-ef7d9b40cd07-kube-api-access-2xm8t\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:37.051874 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:37.051834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:37.054696 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:37.054671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2038113a-1983-4fb3-93f9-ef7d9b40cd07-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6pm2r\" (UID: \"2038113a-1983-4fb3-93f9-ef7d9b40cd07\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:37.234366 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:37.234333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"
Jul 16 20:34:37.366821 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:37.366791    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r"]
Jul 16 20:34:37.370976 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:37.370950    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2038113a_1983_4fb3_93f9_ef7d9b40cd07.slice/crio-2aa7b7ceff6133fe82a640ebe7f9141ea59c4fa3fc8ab51448974e70cdd8491a WatchSource:0}: Error finding container 2aa7b7ceff6133fe82a640ebe7f9141ea59c4fa3fc8ab51448974e70cdd8491a: Status 404 returned error can't find the container with id 2aa7b7ceff6133fe82a640ebe7f9141ea59c4fa3fc8ab51448974e70cdd8491a
Jul 16 20:34:37.957702 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:37.957640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r" event={"ID":"2038113a-1983-4fb3-93f9-ef7d9b40cd07","Type":"ContainerStarted","Data":"2aa7b7ceff6133fe82a640ebe7f9141ea59c4fa3fc8ab51448974e70cdd8491a"}
Jul 16 20:34:44.949043 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:44.949010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lvz59"
Jul 16 20:34:46.893570 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:46.893538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:34:46.994493 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:46.994444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-h6r9r" event={"ID":"30ab91e5-66ba-426d-bd20-e4767ef0202c","Type":"ContainerStarted","Data":"c56c5af6185b999d3826df4bf5a29efe1086c7cafbc6e28976bd6546a2e836b6"}
Jul 16 20:34:46.994975 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:46.994931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-h6r9r"
Jul 16 20:34:46.997098 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:46.997013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r" event={"ID":"2038113a-1983-4fb3-93f9-ef7d9b40cd07","Type":"ContainerStarted","Data":"d57e08c8f8ce32e5050e95b5e01afaa67251f7ac4ac8c554324148c9796a17c1"}
Jul 16 20:34:46.997098 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:46.997063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r" event={"ID":"2038113a-1983-4fb3-93f9-ef7d9b40cd07","Type":"ContainerStarted","Data":"e5d4be4d2f95c1996a9e0b862b43cd948ad3889dfc00300b0b2328996526c7b6"}
Jul 16 20:34:47.012183 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:47.012119    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-h6r9r" podStartSLOduration=3.606382553 podStartE2EDuration="23.012098209s" totalImagesPullingTime="19.405715656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:47.010747518 +0000 UTC m=+94.114026504" watchObservedRunningTime="2026-07-16 20:34:47.012098209 +0000 UTC m=+94.115377193"
Jul 16 20:34:47.019299 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:47.019264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-h6r9r"
Jul 16 20:34:47.033994 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:47.033919    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6pm2r" podStartSLOduration=2.285973258 podStartE2EDuration="11.03390096s" totalImagesPullingTime="8.747927702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:47.032305396 +0000 UTC m=+94.135584385" watchObservedRunningTime="2026-07-16 20:34:47.03390096 +0000 UTC m=+94.137179944"
Jul 16 20:34:48.710412 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.710378    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-qbp74"]
Jul 16 20:34:48.756717 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.756683    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rdrch"]
Jul 16 20:34:48.756882 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.756827    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.760118 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.760085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lng9h\""
Jul 16 20:34:48.760118 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.760085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 16 20:34:48.761239 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.761218    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 16 20:34:48.761239 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.761237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 16 20:34:48.778969 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.778942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-qbp74"]
Jul 16 20:34:48.779095 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.779051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.782861 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.782837    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-trqdw\""
Jul 16 20:34:48.782980 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.782956    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 20:34:48.783483 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.783469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 20:34:48.783726 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.783706    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 20:34:48.854780 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.854745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/04e72e36-5a2a-4975-9075-3c443ac41680-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.854961 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.854790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.854961 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.854832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04e72e36-5a2a-4975-9075-3c443ac41680-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.854961 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.854919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.854961 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.854951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.855139 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.855000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdrm\" (UniqueName: \"kubernetes.io/projected/04e72e36-5a2a-4975-9075-3c443ac41680-kube-api-access-jhdrm\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.956138 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-tls\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956325 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqj7\" (UniqueName: \"kubernetes.io/projected/16b48778-83ea-4905-bc1f-141bd634d46f-kube-api-access-jfqj7\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956325 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/04e72e36-5a2a-4975-9075-3c443ac41680-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.956325 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16b48778-83ea-4905-bc1f-141bd634d46f-metrics-client-ca\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956489 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.956489 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-root\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956489 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956709 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04e72e36-5a2a-4975-9075-3c443ac41680-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.956709 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.956709 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-wtmp\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956709 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/04e72e36-5a2a-4975-9075-3c443ac41680-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.956709 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.956709 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-sys\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-textfile\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:48.956998 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.956802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdrm\" (UniqueName: \"kubernetes.io/projected/04e72e36-5a2a-4975-9075-3c443ac41680-kube-api-access-jhdrm\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.957332 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.957306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04e72e36-5a2a-4975-9075-3c443ac41680-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.957418 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.957326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.959459 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.959435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.959688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.959662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e72e36-5a2a-4975-9075-3c443ac41680-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:48.965890 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:48.965811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdrm\" (UniqueName: \"kubernetes.io/projected/04e72e36-5a2a-4975-9075-3c443ac41680-kube-api-access-jhdrm\") pod \"kube-state-metrics-56c5594674-qbp74\" (UID: \"04e72e36-5a2a-4975-9075-3c443ac41680\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:49.057297 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-wtmp\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057485 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-sys\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057485 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057485 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-textfile\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057485 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-tls\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057485 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqj7\" (UniqueName: \"kubernetes.io/projected/16b48778-83ea-4905-bc1f-141bd634d46f-kube-api-access-jfqj7\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057485 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-sys\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057812 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-wtmp\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057812 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16b48778-83ea-4905-bc1f-141bd634d46f-metrics-client-ca\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057812 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-root\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.057812 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.058018 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.057973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16b48778-83ea-4905-bc1f-141bd634d46f-root\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.058103 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.058052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-textfile\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.058249 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.058226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.058361 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.058340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16b48778-83ea-4905-bc1f-141bd634d46f-metrics-client-ca\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.060093 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.060072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.060462 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.060440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16b48778-83ea-4905-bc1f-141bd634d46f-node-exporter-tls\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.065674 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.065654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqj7\" (UniqueName: \"kubernetes.io/projected/16b48778-83ea-4905-bc1f-141bd634d46f-kube-api-access-jfqj7\") pod \"node-exporter-rdrch\" (UID: \"16b48778-83ea-4905-bc1f-141bd634d46f\") " pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.067549 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.067507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74"
Jul 16 20:34:49.089508 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.089473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rdrch"
Jul 16 20:34:49.129595 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:49.129562    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b48778_83ea_4905_bc1f_141bd634d46f.slice/crio-44761dea36979ebf4bbbfc4f5d352096d6451762e8612442345c977ef7e14bff WatchSource:0}: Error finding container 44761dea36979ebf4bbbfc4f5d352096d6451762e8612442345c977ef7e14bff: Status 404 returned error can't find the container with id 44761dea36979ebf4bbbfc4f5d352096d6451762e8612442345c977ef7e14bff
Jul 16 20:34:49.245591 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:49.245500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-qbp74"]
Jul 16 20:34:49.249541 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:49.249498    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e72e36_5a2a_4975_9075_3c443ac41680.slice/crio-6d0fefeff829182d489bf8d806c2bde5c5a2a15401696ce5f1a40e14df51e5bd WatchSource:0}: Error finding container 6d0fefeff829182d489bf8d806c2bde5c5a2a15401696ce5f1a40e14df51e5bd: Status 404 returned error can't find the container with id 6d0fefeff829182d489bf8d806c2bde5c5a2a15401696ce5f1a40e14df51e5bd
Jul 16 20:34:50.016271 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:50.016224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74" event={"ID":"04e72e36-5a2a-4975-9075-3c443ac41680","Type":"ContainerStarted","Data":"6d0fefeff829182d489bf8d806c2bde5c5a2a15401696ce5f1a40e14df51e5bd"}
Jul 16 20:34:50.018658 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:50.018581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdrch" event={"ID":"16b48778-83ea-4905-bc1f-141bd634d46f","Type":"ContainerStarted","Data":"44761dea36979ebf4bbbfc4f5d352096d6451762e8612442345c977ef7e14bff"}
Jul 16 20:34:51.906902 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:51.906809    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fb765bf46-n2z77" podUID="6db60eec-50b2-4e2c-aec5-3bbc4e981e62" containerName="registry" containerID="cri-o://71f8d66c45e356bbbedbbb14c9d8d57548d0196f1127870388240496da71ff97" gracePeriod=30
Jul 16 20:34:52.028918 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.028882    2580 generic.go:356] "Generic (PLEG): container finished" podID="6db60eec-50b2-4e2c-aec5-3bbc4e981e62" containerID="71f8d66c45e356bbbedbbb14c9d8d57548d0196f1127870388240496da71ff97" exitCode=0
Jul 16 20:34:52.029081 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.028955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb765bf46-n2z77" event={"ID":"6db60eec-50b2-4e2c-aec5-3bbc4e981e62","Type":"ContainerDied","Data":"71f8d66c45e356bbbedbbb14c9d8d57548d0196f1127870388240496da71ff97"}
Jul 16 20:34:52.031396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.031369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74" event={"ID":"04e72e36-5a2a-4975-9075-3c443ac41680","Type":"ContainerStarted","Data":"fc4b8954480a3f6b00fb4841d711f0639ae0054492232d63264e07db3fc1312d"}
Jul 16 20:34:52.031531 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.031421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74" event={"ID":"04e72e36-5a2a-4975-9075-3c443ac41680","Type":"ContainerStarted","Data":"8f6554ffee38f4e8da6aef91ad00bb29dd5accf20beac059ede281dec3b02abd"}
Jul 16 20:34:52.031531 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.031452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74" event={"ID":"04e72e36-5a2a-4975-9075-3c443ac41680","Type":"ContainerStarted","Data":"3da6a19a84ef703c9bc4fd4cd1724918fbb9b99a03971034d89d9f56d2803869"}
Jul 16 20:34:52.033189 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.033159    2580 generic.go:356] "Generic (PLEG): container finished" podID="16b48778-83ea-4905-bc1f-141bd634d46f" containerID="6572e0dc643b68c0b787e55f336962cec3deb5562ffcc45f9a63c0dbe319983f" exitCode=0
Jul 16 20:34:52.033289 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.033191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdrch" event={"ID":"16b48778-83ea-4905-bc1f-141bd634d46f","Type":"ContainerDied","Data":"6572e0dc643b68c0b787e55f336962cec3deb5562ffcc45f9a63c0dbe319983f"}
Jul 16 20:34:52.050360 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.050301    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-qbp74" podStartSLOduration=1.901449973 podStartE2EDuration="4.050283919s" totalImagesPullingTime="2.148833946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:52.048890943 +0000 UTC m=+99.152169945" watchObservedRunningTime="2026-07-16 20:34:52.050283919 +0000 UTC m=+99.153562905"
Jul 16 20:34:52.197420 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.197392    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:34:52.390842 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.390803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-certificates\") pod \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") "
Jul 16 20:34:52.391004 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.390877    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-installation-pull-secrets\") pod \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") "
Jul 16 20:34:52.391004 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.390926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-trusted-ca\") pod \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") "
Jul 16 20:34:52.391004 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.390965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") pod \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") "
Jul 16 20:34:52.391004 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.390993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpbdz\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-kube-api-access-dpbdz\") pod \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") "
Jul 16 20:34:52.391214 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.391020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-ca-trust-extracted\") pod \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") "
Jul 16 20:34:52.391214 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.391052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-image-registry-private-configuration\") pod \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") "
Jul 16 20:34:52.391214 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.391085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-bound-sa-token\") pod \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\" (UID: \"6db60eec-50b2-4e2c-aec5-3bbc4e981e62\") "
Jul 16 20:34:52.391655 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.391392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6db60eec-50b2-4e2c-aec5-3bbc4e981e62" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:34:52.391655 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.391423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6db60eec-50b2-4e2c-aec5-3bbc4e981e62" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:34:52.393915 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.393867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6db60eec-50b2-4e2c-aec5-3bbc4e981e62" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:34:52.394121 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.394094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6db60eec-50b2-4e2c-aec5-3bbc4e981e62" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:34:52.394207 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.394177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6db60eec-50b2-4e2c-aec5-3bbc4e981e62" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:34:52.394406 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.394383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6db60eec-50b2-4e2c-aec5-3bbc4e981e62" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:34:52.394903 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.394877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-kube-api-access-dpbdz" (OuterVolumeSpecName: "kube-api-access-dpbdz") pod "6db60eec-50b2-4e2c-aec5-3bbc4e981e62" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62"). InnerVolumeSpecName "kube-api-access-dpbdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:34:52.403284 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.403247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6db60eec-50b2-4e2c-aec5-3bbc4e981e62" (UID: "6db60eec-50b2-4e2c-aec5-3bbc4e981e62"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:34:52.492528 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.492482    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:34:52.492688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.492546    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpbdz\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-kube-api-access-dpbdz\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:34:52.492688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.492564    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-ca-trust-extracted\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:34:52.492688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.492581    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-image-registry-private-configuration\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:34:52.492688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.492596    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-bound-sa-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:34:52.492688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.492610    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-registry-certificates\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:34:52.492688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.492624    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-installation-pull-secrets\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:34:52.492688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:52.492638    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6db60eec-50b2-4e2c-aec5-3bbc4e981e62-trusted-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:34:53.040302 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.040239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdrch" event={"ID":"16b48778-83ea-4905-bc1f-141bd634d46f","Type":"ContainerStarted","Data":"42317a58b15074d08a6bae17c71e3c3a950a440b37d83f1a23b51db88e546d6d"}
Jul 16 20:34:53.040779 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.040320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdrch" event={"ID":"16b48778-83ea-4905-bc1f-141bd634d46f","Type":"ContainerStarted","Data":"bb40ec962175a0a9c5899a7fbafa9d874ab55cd1a7712c01ae30faf06fff2471"}
Jul 16 20:34:53.041761 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.041735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb765bf46-n2z77" event={"ID":"6db60eec-50b2-4e2c-aec5-3bbc4e981e62","Type":"ContainerDied","Data":"603774c47f3bb8e4f88a2b3d59bdd827c611cc0e63f22a1e3557727730ccb00f"}
Jul 16 20:34:53.041871 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.041770    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb765bf46-n2z77"
Jul 16 20:34:53.041871 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.041814    2580 scope.go:117] "RemoveContainer" containerID="71f8d66c45e356bbbedbbb14c9d8d57548d0196f1127870388240496da71ff97"
Jul 16 20:34:53.061554 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.061464    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rdrch" podStartSLOduration=2.669972247 podStartE2EDuration="5.061441114s" totalImagesPullingTime="2.26656348s" totalInitContainerRuntime="124.905387ms" isStatefulPod=true podCreationTimestamp="2026-07-16 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:53.059302484 +0000 UTC m=+100.162581517" watchObservedRunningTime="2026-07-16 20:34:53.061441114 +0000 UTC m=+100.164720099"
Jul 16 20:34:53.074440 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.074404    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb765bf46-n2z77"]
Jul 16 20:34:53.078692 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.078665    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fb765bf46-n2z77"]
Jul 16 20:34:53.481332 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.480346    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"]
Jul 16 20:34:53.481332 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.480736    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db60eec-50b2-4e2c-aec5-3bbc4e981e62" containerName="registry"
Jul 16 20:34:53.481332 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.480753    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db60eec-50b2-4e2c-aec5-3bbc4e981e62" containerName="registry"
Jul 16 20:34:53.481332 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.480813    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db60eec-50b2-4e2c-aec5-3bbc4e981e62" containerName="registry"
Jul 16 20:34:53.505944 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.505907    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"]
Jul 16 20:34:53.506134 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.506113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:53.508621 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.508597    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 16 20:34:53.508765 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.508660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-stbr2\""
Jul 16 20:34:53.512116 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.512088    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db60eec-50b2-4e2c-aec5-3bbc4e981e62" path="/var/lib/kubelet/pods/6db60eec-50b2-4e2c-aec5-3bbc4e981e62/volumes"
Jul 16 20:34:53.601703 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.601670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-4l9jv\" (UID: \"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:53.601892 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.601707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdbk\" (UniqueName: \"kubernetes.io/projected/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-kube-api-access-lpdbk\") pod \"monitoring-plugin-7b7b7bc8bf-4l9jv\" (UID: \"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:53.702489 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.702453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-4l9jv\" (UID: \"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:53.702719 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.702503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdbk\" (UniqueName: \"kubernetes.io/projected/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-kube-api-access-lpdbk\") pod \"monitoring-plugin-7b7b7bc8bf-4l9jv\" (UID: \"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:53.702719 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:53.702650    2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 16 20:34:53.702828 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:34:53.702727    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-monitoring-plugin-cert podName:1b73f3dc-7f24-4aa8-b94b-bc0498ec898d nodeName:}" failed. No retries permitted until 2026-07-16 20:34:54.202704843 +0000 UTC m=+101.305983810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-4l9jv" (UID: "1b73f3dc-7f24-4aa8-b94b-bc0498ec898d") : secret "monitoring-plugin-cert" not found
Jul 16 20:34:53.711561 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:53.711537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdbk\" (UniqueName: \"kubernetes.io/projected/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-kube-api-access-lpdbk\") pod \"monitoring-plugin-7b7b7bc8bf-4l9jv\" (UID: \"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:54.207017 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.206981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-4l9jv\" (UID: \"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:54.209653 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.209628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b73f3dc-7f24-4aa8-b94b-bc0498ec898d-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-4l9jv\" (UID: \"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:54.416940 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.416904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:54.620345 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.620313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"]
Jul 16 20:34:54.623701 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:54.623665    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b73f3dc_7f24_4aa8_b94b_bc0498ec898d.slice/crio-48b6d13c473b0dc0c05afb2d4b78cac8cddcbb9e9d9e4f01f26bbd01cedc79a8 WatchSource:0}: Error finding container 48b6d13c473b0dc0c05afb2d4b78cac8cddcbb9e9d9e4f01f26bbd01cedc79a8: Status 404 returned error can't find the container with id 48b6d13c473b0dc0c05afb2d4b78cac8cddcbb9e9d9e4f01f26bbd01cedc79a8
Jul 16 20:34:54.957364 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.957324    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:34:54.988662 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.988626    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:34:54.988852 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.988780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:54.993156 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.992670    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 20:34:54.993156 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.992721    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 20:34:54.993156 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.992673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a91snb6ksbmj\""
Jul 16 20:34:54.993156 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.992986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 20:34:54.993156 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.993084    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 20:34:54.993156 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.993117    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 20:34:54.993567 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.993228    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 20:34:54.993567 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.993327    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 20:34:54.995164 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.995132    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 20:34:54.996545 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.995635    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 20:34:54.996764 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.995735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 20:34:54.996989 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.996956    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 20:34:54.997453 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.997205    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-znf8s\""
Jul 16 20:34:54.997453 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:54.997405    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 20:34:55.000719 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.000220    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 20:34:55.001565 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.001372    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 20:34:55.002349 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.002218    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 20:34:55.012447 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.012415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.012638 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.012459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-config\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.012638 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.012508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e8da451-de2f-4eeb-8d17-176047e38b4d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.012638 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.012596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e8da451-de2f-4eeb-8d17-176047e38b4d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.012791 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.012640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.052273 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.051927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv" event={"ID":"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d","Type":"ContainerStarted","Data":"48b6d13c473b0dc0c05afb2d4b78cac8cddcbb9e9d9e4f01f26bbd01cedc79a8"}
Jul 16 20:34:55.113655 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.113655 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.113878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkznz\" (UniqueName: \"kubernetes.io/projected/5e8da451-de2f-4eeb-8d17-176047e38b4d-kube-api-access-dkznz\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.113878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.113878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.113878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.113878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.113878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-config\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.113878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e8da451-de2f-4eeb-8d17-176047e38b4d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.113983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.114024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e8da451-de2f-4eeb-8d17-176047e38b4d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.114053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.114077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.114101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.114146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114181 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.114181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.114670 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.114205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.116937 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.116913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e8da451-de2f-4eeb-8d17-176047e38b4d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.117247 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.117132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.117247 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.117157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e8da451-de2f-4eeb-8d17-176047e38b4d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.117247 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.117186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.117247 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.117202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-config\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.215262 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.215262 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.215262 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkznz\" (UniqueName: \"kubernetes.io/projected/5e8da451-de2f-4eeb-8d17-176047e38b4d-kube-api-access-dkznz\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.215995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.216049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.216280 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.216278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.218874 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.217202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.218874 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.217606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.218874 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.217894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.218874 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.218093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.220178 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.219888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.220295 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.220235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.220783 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.220750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.221480 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.221447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.221480 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.221471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e8da451-de2f-4eeb-8d17-176047e38b4d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.221684 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.221488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.223581 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.223545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e8da451-de2f-4eeb-8d17-176047e38b4d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.226125 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.226082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkznz\" (UniqueName: \"kubernetes.io/projected/5e8da451-de2f-4eeb-8d17-176047e38b4d-kube-api-access-dkznz\") pod \"prometheus-k8s-0\" (UID: \"5e8da451-de2f-4eeb-8d17-176047e38b4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.307932 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.307890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:34:55.493930 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:55.493851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:34:55.501178 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:34:55.501149    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8da451_de2f_4eeb_8d17_176047e38b4d.slice/crio-863b9369a49662384e3f2549bccfbf53b5e8dd66a381ccddd669492fa8cebfad WatchSource:0}: Error finding container 863b9369a49662384e3f2549bccfbf53b5e8dd66a381ccddd669492fa8cebfad: Status 404 returned error can't find the container with id 863b9369a49662384e3f2549bccfbf53b5e8dd66a381ccddd669492fa8cebfad
Jul 16 20:34:56.057060 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:56.057017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e8da451-de2f-4eeb-8d17-176047e38b4d","Type":"ContainerStarted","Data":"863b9369a49662384e3f2549bccfbf53b5e8dd66a381ccddd669492fa8cebfad"}
Jul 16 20:34:57.901176 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:57.901146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z4nkd"
Jul 16 20:34:58.065444 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:58.065408    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e8da451-de2f-4eeb-8d17-176047e38b4d" containerID="c73ff6566eb1498d74fbcb12d518eb9f06cc4965377d147d0e902468fbd3b629" exitCode=0
Jul 16 20:34:58.065646 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:58.065505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e8da451-de2f-4eeb-8d17-176047e38b4d","Type":"ContainerDied","Data":"c73ff6566eb1498d74fbcb12d518eb9f06cc4965377d147d0e902468fbd3b629"}
Jul 16 20:34:58.067033 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:58.067001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv" event={"ID":"1b73f3dc-7f24-4aa8-b94b-bc0498ec898d","Type":"ContainerStarted","Data":"49213dd9c43476602a41f51f3fb218e70f112810aa7adfb1ca2d082e09a6e9e2"}
Jul 16 20:34:58.067278 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:58.067264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:58.072622 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:58.072598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv"
Jul 16 20:34:58.132240 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:34:58.132194    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4l9jv" podStartSLOduration=2.698710362 podStartE2EDuration="5.132177421s" totalImagesPullingTime="2.433467059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:58.130813317 +0000 UTC m=+105.234092321" watchObservedRunningTime="2026-07-16 20:34:58.132177421 +0000 UTC m=+105.235456407"
Jul 16 20:35:02.089967 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:02.089931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e8da451-de2f-4eeb-8d17-176047e38b4d","Type":"ContainerStarted","Data":"f87c44eac081668a6847e6e0fa1e1eacb9ebafe32f082e3e5857bb86fc4e176f"}
Jul 16 20:35:03.096493 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:03.096456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e8da451-de2f-4eeb-8d17-176047e38b4d","Type":"ContainerStarted","Data":"f1d56c753f2ae23d56fa02ef55d974f6455de7c42fe504c7339f1604deef9aa4"}
Jul 16 20:35:05.114168 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:05.114141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e8da451-de2f-4eeb-8d17-176047e38b4d","Type":"ContainerStarted","Data":"389473800e6994c00863347abf2a6532a9de878e2e4fa1ea955408c33061ef90"}
Jul 16 20:35:05.114599 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:05.114227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e8da451-de2f-4eeb-8d17-176047e38b4d","Type":"ContainerStarted","Data":"dddb719ec0a84f9e6b1b3d084a287845df90acb8cb9ff4e828e0645aff5b23c1"}
Jul 16 20:35:05.114599 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:05.114253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e8da451-de2f-4eeb-8d17-176047e38b4d","Type":"ContainerStarted","Data":"448c1b9ee2995a41571f314c05ae4259c220eb0ffe72255fb7a42608726f125f"}
Jul 16 20:35:06.120863 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:06.120823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e8da451-de2f-4eeb-8d17-176047e38b4d","Type":"ContainerStarted","Data":"2dafcf2cafba3b40c6d99b8c1ab58d63b0809d054fdc99c4af863a4eef4edbed"}
Jul 16 20:35:06.151042 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:06.150992    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.912796158 podStartE2EDuration="12.150975788s" totalImagesPullingTime="8.212971728s" totalInitContainerRuntime="25.207902ms" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:35:06.149907229 +0000 UTC m=+113.253186233" watchObservedRunningTime="2026-07-16 20:35:06.150975788 +0000 UTC m=+113.254254773"
Jul 16 20:35:10.308626 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:10.308593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:35:55.309002 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:55.308970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:35:55.328228 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:55.328192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:35:56.321355 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:35:56.321329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:38:13.299718 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:13.299686    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:38:13.301602 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:13.301581    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:38:13.302670 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:13.302653    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 20:38:34.344098 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.344063    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"]
Jul 16 20:38:34.346252 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.346235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.348836 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.348812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 20:38:34.349246 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.349232    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 20:38:34.350375 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.350350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 20:38:34.350375 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.350362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vwh9v\""
Jul 16 20:38:34.350548 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.350368    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 20:38:34.372114 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.372093    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"]
Jul 16 20:38:34.429908 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.429870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77eeac63-d6a2-4459-9ac8-8af8351ddd02-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.429908 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.429908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95cm\" (UniqueName: \"kubernetes.io/projected/77eeac63-d6a2-4459-9ac8-8af8351ddd02-kube-api-access-v95cm\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.430099 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.429972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77eeac63-d6a2-4459-9ac8-8af8351ddd02-tmp\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.430099 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.429994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77eeac63-d6a2-4459-9ac8-8af8351ddd02-manifests\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.430099 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.430020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77eeac63-d6a2-4459-9ac8-8af8351ddd02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.530642 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.530603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77eeac63-d6a2-4459-9ac8-8af8351ddd02-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.530642 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.530637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95cm\" (UniqueName: \"kubernetes.io/projected/77eeac63-d6a2-4459-9ac8-8af8351ddd02-kube-api-access-v95cm\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.530853 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.530777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77eeac63-d6a2-4459-9ac8-8af8351ddd02-tmp\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.530853 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.530816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77eeac63-d6a2-4459-9ac8-8af8351ddd02-manifests\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.530853 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.530848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77eeac63-d6a2-4459-9ac8-8af8351ddd02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.531187 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.531158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77eeac63-d6a2-4459-9ac8-8af8351ddd02-manifests\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.533057 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.533033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77eeac63-d6a2-4459-9ac8-8af8351ddd02-tmp\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.533162 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.533147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77eeac63-d6a2-4459-9ac8-8af8351ddd02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.533253 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.533239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77eeac63-d6a2-4459-9ac8-8af8351ddd02-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.539157 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.539139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95cm\" (UniqueName: \"kubernetes.io/projected/77eeac63-d6a2-4459-9ac8-8af8351ddd02-kube-api-access-v95cm\") pod \"opendatahub-operator-controller-manager-6d6477549b-4t5tr\" (UID: \"77eeac63-d6a2-4459-9ac8-8af8351ddd02\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.656011 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.655983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:34.820334 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.820303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"]
Jul 16 20:38:34.824815 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:38:34.824787    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77eeac63_d6a2_4459_9ac8_8af8351ddd02.slice/crio-e553b62433048458b3c8c59769814ced3b61401d58ad9c662ac43bbce8153fa2 WatchSource:0}: Error finding container e553b62433048458b3c8c59769814ced3b61401d58ad9c662ac43bbce8153fa2: Status 404 returned error can't find the container with id e553b62433048458b3c8c59769814ced3b61401d58ad9c662ac43bbce8153fa2
Jul 16 20:38:34.826686 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.826667    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:38:34.858150 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:34.858116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr" event={"ID":"77eeac63-d6a2-4459-9ac8-8af8351ddd02","Type":"ContainerStarted","Data":"e553b62433048458b3c8c59769814ced3b61401d58ad9c662ac43bbce8153fa2"}
Jul 16 20:38:39.880723 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:39.880689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr" event={"ID":"77eeac63-d6a2-4459-9ac8-8af8351ddd02","Type":"ContainerStarted","Data":"fdd4cae819db456f85965827200d57e8d20f77e428115843eb2bd3653f2e0e16"}
Jul 16 20:38:40.886421 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:40.886384    2580 generic.go:356] "Generic (PLEG): container finished" podID="77eeac63-d6a2-4459-9ac8-8af8351ddd02" containerID="fdd4cae819db456f85965827200d57e8d20f77e428115843eb2bd3653f2e0e16" exitCode=0
Jul 16 20:38:40.886421 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:40.886423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr" event={"ID":"77eeac63-d6a2-4459-9ac8-8af8351ddd02","Type":"ContainerDied","Data":"fdd4cae819db456f85965827200d57e8d20f77e428115843eb2bd3653f2e0e16"}
Jul 16 20:38:41.892704 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:41.892672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr" event={"ID":"77eeac63-d6a2-4459-9ac8-8af8351ddd02","Type":"ContainerStarted","Data":"6411b2aa92e080937834c358c8ac585f7d9d9c537bf7e8920dfababeec212c94"}
Jul 16 20:38:41.893088 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:41.892929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:41.919003 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:41.918948    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr" podStartSLOduration=2.292071838 podStartE2EDuration="7.91893388s" totalImagesPullingTime="4.599112492s" totalInitContainerRuntime="1.02774955s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:38:41.917330695 +0000 UTC m=+329.020609706" watchObservedRunningTime="2026-07-16 20:38:41.91893388 +0000 UTC m=+329.022212864"
Jul 16 20:38:52.899385 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:52.899343    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-4t5tr"
Jul 16 20:38:54.293232 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.293199    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"]
Jul 16 20:38:54.302589 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.302568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.306650 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.306627    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 20:38:54.306795 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.306631    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 20:38:54.306795 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.306698    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-97rsl\""
Jul 16 20:38:54.306795 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.306712    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 20:38:54.306940 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.306924    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 20:38:54.306974 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.306924    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 20:38:54.312835 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.312811    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"]
Jul 16 20:38:54.390939 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.390915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28cbdb0a-9644-497e-aa48-78f2f020340f-manager-config\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.391100 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.390962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28cbdb0a-9644-497e-aa48-78f2f020340f-metrics-cert\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.391100 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.391062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28cbdb0a-9644-497e-aa48-78f2f020340f-cert\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.391211 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.391103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkd4m\" (UniqueName: \"kubernetes.io/projected/28cbdb0a-9644-497e-aa48-78f2f020340f-kube-api-access-lkd4m\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.492540 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.492485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28cbdb0a-9644-497e-aa48-78f2f020340f-cert\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.492540 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.492541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkd4m\" (UniqueName: \"kubernetes.io/projected/28cbdb0a-9644-497e-aa48-78f2f020340f-kube-api-access-lkd4m\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.492774 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.492582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28cbdb0a-9644-497e-aa48-78f2f020340f-manager-config\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.492774 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.492617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28cbdb0a-9644-497e-aa48-78f2f020340f-metrics-cert\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.493282 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.493259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28cbdb0a-9644-497e-aa48-78f2f020340f-manager-config\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.494952 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.494921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28cbdb0a-9644-497e-aa48-78f2f020340f-cert\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.495058 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.495008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28cbdb0a-9644-497e-aa48-78f2f020340f-metrics-cert\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.530404 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.530375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkd4m\" (UniqueName: \"kubernetes.io/projected/28cbdb0a-9644-497e-aa48-78f2f020340f-kube-api-access-lkd4m\") pod \"lws-controller-manager-649bd55f77-ffth9\" (UID: \"28cbdb0a-9644-497e-aa48-78f2f020340f\") " pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.611294 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.611218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:38:54.766763 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.766736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"]
Jul 16 20:38:54.769676 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:38:54.769650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cbdb0a_9644_497e_aa48_78f2f020340f.slice/crio-692630d3145e2f69fe92705d06611199c8651b48223960b953d0d8d53e48d6d2 WatchSource:0}: Error finding container 692630d3145e2f69fe92705d06611199c8651b48223960b953d0d8d53e48d6d2: Status 404 returned error can't find the container with id 692630d3145e2f69fe92705d06611199c8651b48223960b953d0d8d53e48d6d2
Jul 16 20:38:54.943927 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:54.943896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9" event={"ID":"28cbdb0a-9644-497e-aa48-78f2f020340f","Type":"ContainerStarted","Data":"692630d3145e2f69fe92705d06611199c8651b48223960b953d0d8d53e48d6d2"}
Jul 16 20:38:55.282574 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.282493    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"]
Jul 16 20:38:55.285225 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.285205    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.288781 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.288757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 20:38:55.288781 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.288768    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 20:38:55.289653 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.289622    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 20:38:55.289753 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.289663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 20:38:55.290293 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.290280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mb988\""
Jul 16 20:38:55.297381 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.297358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"]
Jul 16 20:38:55.398870 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.398819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddqg8\" (UniqueName: \"kubernetes.io/projected/139ab020-1707-4640-91b8-43b3e04d5e0f-kube-api-access-ddqg8\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.399064 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.398906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/139ab020-1707-4640-91b8-43b3e04d5e0f-tmp\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.399064 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.399033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139ab020-1707-4640-91b8-43b3e04d5e0f-tls-certs\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.499960 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.499929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139ab020-1707-4640-91b8-43b3e04d5e0f-tls-certs\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.500136 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.499973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddqg8\" (UniqueName: \"kubernetes.io/projected/139ab020-1707-4640-91b8-43b3e04d5e0f-kube-api-access-ddqg8\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.500136 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.499998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/139ab020-1707-4640-91b8-43b3e04d5e0f-tmp\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.502340 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.502314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/139ab020-1707-4640-91b8-43b3e04d5e0f-tmp\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.502660 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.502641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139ab020-1707-4640-91b8-43b3e04d5e0f-tls-certs\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.508491 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.508472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddqg8\" (UniqueName: \"kubernetes.io/projected/139ab020-1707-4640-91b8-43b3e04d5e0f-kube-api-access-ddqg8\") pod \"kube-auth-proxy-559f5f697c-jlhr5\" (UID: \"139ab020-1707-4640-91b8-43b3e04d5e0f\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.595691 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.595603    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"
Jul 16 20:38:55.733681 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.733651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5"]
Jul 16 20:38:55.737020 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:38:55.736996    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139ab020_1707_4640_91b8_43b3e04d5e0f.slice/crio-a24343620b82a9a18d8b85ee71c4801c4e69c18e1c3fe9ad9d3ac8ca0b71562b WatchSource:0}: Error finding container a24343620b82a9a18d8b85ee71c4801c4e69c18e1c3fe9ad9d3ac8ca0b71562b: Status 404 returned error can't find the container with id a24343620b82a9a18d8b85ee71c4801c4e69c18e1c3fe9ad9d3ac8ca0b71562b
Jul 16 20:38:55.949388 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:55.949346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5" event={"ID":"139ab020-1707-4640-91b8-43b3e04d5e0f","Type":"ContainerStarted","Data":"a24343620b82a9a18d8b85ee71c4801c4e69c18e1c3fe9ad9d3ac8ca0b71562b"}
Jul 16 20:38:59.969362 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:38:59.969323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5" event={"ID":"139ab020-1707-4640-91b8-43b3e04d5e0f","Type":"ContainerStarted","Data":"1dbda08534ae16072d983a2378e996c5ab50582a7b3e198fba4cefbd5123ca3e"}
Jul 16 20:39:00.011458 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:00.011409    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-559f5f697c-jlhr5" podStartSLOduration=1.591247227 podStartE2EDuration="5.011391712s" totalImagesPullingTime="3.420144485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:00.010989654 +0000 UTC m=+347.114268641" watchObservedRunningTime="2026-07-16 20:39:00.011391712 +0000 UTC m=+347.114670697"
Jul 16 20:39:01.979570 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:01.979482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9" event={"ID":"28cbdb0a-9644-497e-aa48-78f2f020340f","Type":"ContainerStarted","Data":"c77bfae39e91685338cf7b9f39e275786b33da2083e2c4365779f627e829b6a6"}
Jul 16 20:39:01.980015 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:01.979819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:39:02.016452 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:02.016401    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9" podStartSLOduration=0.966482632 podStartE2EDuration="8.01638637s" totalImagesPullingTime="7.049903738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:02.013791698 +0000 UTC m=+349.117070702" watchObservedRunningTime="2026-07-16 20:39:02.01638637 +0000 UTC m=+349.119665354"
Jul 16 20:39:08.273793 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.273708    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"]
Jul 16 20:39:08.277101 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.277084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:08.279954 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.279934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 16 20:39:08.285576 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.285555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kpw24\""
Jul 16 20:39:08.331271 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.331239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"]
Jul 16 20:39:08.405854 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.405823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d924c48-7c85-44fe-b87d-dea94fa4a233-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l4972\" (UID: \"1d924c48-7c85-44fe-b87d-dea94fa4a233\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:08.406025 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.405878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtksd\" (UniqueName: \"kubernetes.io/projected/1d924c48-7c85-44fe-b87d-dea94fa4a233-kube-api-access-rtksd\") pod \"llmisvc-controller-manager-777c6bd7d5-l4972\" (UID: \"1d924c48-7c85-44fe-b87d-dea94fa4a233\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:08.506874 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.506847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtksd\" (UniqueName: \"kubernetes.io/projected/1d924c48-7c85-44fe-b87d-dea94fa4a233-kube-api-access-rtksd\") pod \"llmisvc-controller-manager-777c6bd7d5-l4972\" (UID: \"1d924c48-7c85-44fe-b87d-dea94fa4a233\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:08.507040 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.506920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d924c48-7c85-44fe-b87d-dea94fa4a233-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l4972\" (UID: \"1d924c48-7c85-44fe-b87d-dea94fa4a233\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:08.507040 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:39:08.507023    2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 16 20:39:08.507139 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:39:08.507126    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d924c48-7c85-44fe-b87d-dea94fa4a233-cert podName:1d924c48-7c85-44fe-b87d-dea94fa4a233 nodeName:}" failed. No retries permitted until 2026-07-16 20:39:09.007105176 +0000 UTC m=+356.110384152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1d924c48-7c85-44fe-b87d-dea94fa4a233-cert") pod "llmisvc-controller-manager-777c6bd7d5-l4972" (UID: "1d924c48-7c85-44fe-b87d-dea94fa4a233") : secret "llmisvc-webhook-server-cert" not found
Jul 16 20:39:08.525373 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:08.525300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtksd\" (UniqueName: \"kubernetes.io/projected/1d924c48-7c85-44fe-b87d-dea94fa4a233-kube-api-access-rtksd\") pod \"llmisvc-controller-manager-777c6bd7d5-l4972\" (UID: \"1d924c48-7c85-44fe-b87d-dea94fa4a233\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:09.012437 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:09.012402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d924c48-7c85-44fe-b87d-dea94fa4a233-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l4972\" (UID: \"1d924c48-7c85-44fe-b87d-dea94fa4a233\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:09.015033 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:09.015014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d924c48-7c85-44fe-b87d-dea94fa4a233-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l4972\" (UID: \"1d924c48-7c85-44fe-b87d-dea94fa4a233\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:09.186838 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:09.186795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:09.336499 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:09.336474    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"]
Jul 16 20:39:09.339191 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:39:09.339166    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1d924c48_7c85_44fe_b87d_dea94fa4a233.slice/crio-42c0795eaf68fd71f8772848450623940e85440d87c9041473aaa72a2f5382f7 WatchSource:0}: Error finding container 42c0795eaf68fd71f8772848450623940e85440d87c9041473aaa72a2f5382f7: Status 404 returned error can't find the container with id 42c0795eaf68fd71f8772848450623940e85440d87c9041473aaa72a2f5382f7
Jul 16 20:39:10.014360 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:10.014325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972" event={"ID":"1d924c48-7c85-44fe-b87d-dea94fa4a233","Type":"ContainerStarted","Data":"42c0795eaf68fd71f8772848450623940e85440d87c9041473aaa72a2f5382f7"}
Jul 16 20:39:12.024232 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:12.024140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972" event={"ID":"1d924c48-7c85-44fe-b87d-dea94fa4a233","Type":"ContainerStarted","Data":"ca508b95786468c90bfb470b47dba406df8eaf09625e209e134a1b09f87c4b9b"}
Jul 16 20:39:12.024688 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:12.024249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:39:12.047657 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:12.047610    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972" podStartSLOduration=1.635772285 podStartE2EDuration="4.047592162s" totalImagesPullingTime="2.411819877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:12.04645924 +0000 UTC m=+359.149738253" watchObservedRunningTime="2026-07-16 20:39:12.047592162 +0000 UTC m=+359.150871165"
Jul 16 20:39:12.986036 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:12.986004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-649bd55f77-ffth9"
Jul 16 20:39:43.031648 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:39:43.031619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4972"
Jul 16 20:40:37.777658 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.777582    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"]
Jul 16 20:40:37.780386 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.780371    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.784456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.784434    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 16 20:40:37.784456 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.784450    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 20:40:37.784639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.784467    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2rp5w\""
Jul 16 20:40:37.785683 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.785666    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 16 20:40:37.785768 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.785664    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 20:40:37.794415 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.794394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"]
Jul 16 20:40:37.850561 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.850529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhpj\" (UniqueName: \"kubernetes.io/projected/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-kube-api-access-gkhpj\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.850718 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.850597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.850718 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.850618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.951883 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.951850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.951883 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.951883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.952134 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.951930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhpj\" (UniqueName: \"kubernetes.io/projected/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-kube-api-access-gkhpj\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.952681 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.952660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.954234 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.954213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:37.961170 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:37.961143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhpj\" (UniqueName: \"kubernetes.io/projected/4e423712-9fa3-4d1b-8c2b-32ab61aba48b-kube-api-access-gkhpj\") pod \"kuadrant-console-plugin-66b5774985-qscrv\" (UID: \"4e423712-9fa3-4d1b-8c2b-32ab61aba48b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:38.089660 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:38.089579    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"
Jul 16 20:40:38.228765 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:38.228736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv"]
Jul 16 20:40:38.232357 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:40:38.232329    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e423712_9fa3_4d1b_8c2b_32ab61aba48b.slice/crio-f9d450424f35ac2604ea4f288e737e75cabf4923d155090a4136d088e3aa1998 WatchSource:0}: Error finding container f9d450424f35ac2604ea4f288e737e75cabf4923d155090a4136d088e3aa1998: Status 404 returned error can't find the container with id f9d450424f35ac2604ea4f288e737e75cabf4923d155090a4136d088e3aa1998
Jul 16 20:40:38.383807 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:40:38.383775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv" event={"ID":"4e423712-9fa3-4d1b-8c2b-32ab61aba48b","Type":"ContainerStarted","Data":"f9d450424f35ac2604ea4f288e737e75cabf4923d155090a4136d088e3aa1998"}
Jul 16 20:41:01.498502 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:01.498471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv" event={"ID":"4e423712-9fa3-4d1b-8c2b-32ab61aba48b","Type":"ContainerStarted","Data":"54570059f3ff8343b01a2e462a0d87111c0f67352ccc7a69a4e5a672ab7b439f"}
Jul 16 20:41:01.521469 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:01.521410    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qscrv" podStartSLOduration=1.4803200269999999 podStartE2EDuration="24.521394465s" totalImagesPullingTime="23.041074438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:01.520051079 +0000 UTC m=+468.623330093" watchObservedRunningTime="2026-07-16 20:41:01.521394465 +0000 UTC m=+468.624673450"
Jul 16 20:41:20.235899 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.235867    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"]
Jul 16 20:41:20.364641 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.364607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"]
Jul 16 20:41:20.364827 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.364655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"]
Jul 16 20:41:20.364827 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.364755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:20.367286 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.367263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 20:41:20.521687 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.521579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czjx\" (UniqueName: \"kubernetes.io/projected/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-kube-api-access-5czjx\") pod \"limitador-limitador-64c94fd64f-k6vd8\" (UID: \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:20.521687 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.521646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-config-file\") pod \"limitador-limitador-64c94fd64f-k6vd8\" (UID: \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:20.622878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.622836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czjx\" (UniqueName: \"kubernetes.io/projected/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-kube-api-access-5czjx\") pod \"limitador-limitador-64c94fd64f-k6vd8\" (UID: \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:20.623074 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.623030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-config-file\") pod \"limitador-limitador-64c94fd64f-k6vd8\" (UID: \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:20.623694 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.623674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-config-file\") pod \"limitador-limitador-64c94fd64f-k6vd8\" (UID: \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:20.632357 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.632328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czjx\" (UniqueName: \"kubernetes.io/projected/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-kube-api-access-5czjx\") pod \"limitador-limitador-64c94fd64f-k6vd8\" (UID: \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:20.674624 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.674594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:20.820343 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:20.820319    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"]
Jul 16 20:41:20.822925 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:41:20.822901    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d83f14f_86cc_4dcc_bff6_8fc9e2308bda.slice/crio-eb7d7f32cdf1f1c9b15bd9f701006d0d5c0108502b077469cc7a43aeda9fea25 WatchSource:0}: Error finding container eb7d7f32cdf1f1c9b15bd9f701006d0d5c0108502b077469cc7a43aeda9fea25: Status 404 returned error can't find the container with id eb7d7f32cdf1f1c9b15bd9f701006d0d5c0108502b077469cc7a43aeda9fea25
Jul 16 20:41:21.576425 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:21.576391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8" event={"ID":"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda","Type":"ContainerStarted","Data":"eb7d7f32cdf1f1c9b15bd9f701006d0d5c0108502b077469cc7a43aeda9fea25"}
Jul 16 20:41:25.604917 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:25.604880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8" event={"ID":"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda","Type":"ContainerStarted","Data":"42d1414af2b38dc99eef9b62d00894326bb9f8cd50b20794de5491261f122e50"}
Jul 16 20:41:25.605396 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:25.605008    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:25.622043 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:25.621975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8" podStartSLOduration=1.9219371189999999 podStartE2EDuration="5.621962305s" totalImagesPullingTime="3.700025186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:25.621027114 +0000 UTC m=+492.724306100" watchObservedRunningTime="2026-07-16 20:41:25.621962305 +0000 UTC m=+492.725241342"
Jul 16 20:41:35.109607 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.109575    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"]
Jul 16 20:41:35.110021 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.109754    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8" podUID="7d83f14f-86cc-4dcc-bff6-8fc9e2308bda" containerName="limitador" containerID="cri-o://42d1414af2b38dc99eef9b62d00894326bb9f8cd50b20794de5491261f122e50" gracePeriod=30
Jul 16 20:41:35.110435 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.110389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:35.651725 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.651696    2580 generic.go:356] "Generic (PLEG): container finished" podID="7d83f14f-86cc-4dcc-bff6-8fc9e2308bda" containerID="42d1414af2b38dc99eef9b62d00894326bb9f8cd50b20794de5491261f122e50" exitCode=0
Jul 16 20:41:35.651877 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.651762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8" event={"ID":"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda","Type":"ContainerDied","Data":"42d1414af2b38dc99eef9b62d00894326bb9f8cd50b20794de5491261f122e50"}
Jul 16 20:41:35.697439 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.697413    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:35.746930 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.746901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5czjx\" (UniqueName: \"kubernetes.io/projected/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-kube-api-access-5czjx\") pod \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\" (UID: \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\") "
Jul 16 20:41:35.747179 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.746935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-config-file\") pod \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\" (UID: \"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda\") "
Jul 16 20:41:35.747374 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.747345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-config-file" (OuterVolumeSpecName: "config-file") pod "7d83f14f-86cc-4dcc-bff6-8fc9e2308bda" (UID: "7d83f14f-86cc-4dcc-bff6-8fc9e2308bda"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:41:35.749185 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.749165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-kube-api-access-5czjx" (OuterVolumeSpecName: "kube-api-access-5czjx") pod "7d83f14f-86cc-4dcc-bff6-8fc9e2308bda" (UID: "7d83f14f-86cc-4dcc-bff6-8fc9e2308bda"). InnerVolumeSpecName "kube-api-access-5czjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:41:35.848126 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.848092    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5czjx\" (UniqueName: \"kubernetes.io/projected/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-kube-api-access-5czjx\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:41:35.848126 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:35.848121    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda-config-file\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:41:36.656855 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.656819    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"
Jul 16 20:41:36.656855 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.656831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6vd8" event={"ID":"7d83f14f-86cc-4dcc-bff6-8fc9e2308bda","Type":"ContainerDied","Data":"eb7d7f32cdf1f1c9b15bd9f701006d0d5c0108502b077469cc7a43aeda9fea25"}
Jul 16 20:41:36.657483 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.656885    2580 scope.go:117] "RemoveContainer" containerID="42d1414af2b38dc99eef9b62d00894326bb9f8cd50b20794de5491261f122e50"
Jul 16 20:41:36.686217 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.686192    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"]
Jul 16 20:41:36.705575 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.705553    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6vd8"]
Jul 16 20:41:36.835464 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.835429    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"]
Jul 16 20:41:36.835792 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.835779    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d83f14f-86cc-4dcc-bff6-8fc9e2308bda" containerName="limitador"
Jul 16 20:41:36.835792 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.835793    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d83f14f-86cc-4dcc-bff6-8fc9e2308bda" containerName="limitador"
Jul 16 20:41:36.835878 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.835843    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d83f14f-86cc-4dcc-bff6-8fc9e2308bda" containerName="limitador"
Jul 16 20:41:36.838981 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.838961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:36.841656 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.841639    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-clq69\""
Jul 16 20:41:36.842057 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.842042    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 16 20:41:36.842267 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.842251    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 20:41:36.842407 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.842302    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 20:41:36.847986 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.847968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"]
Jul 16 20:41:36.856822 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.856802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca-data\") pod \"postgres-69fb4594cf-wwb25\" (UID: \"6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:36.856913 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.856834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm2h7\" (UniqueName: \"kubernetes.io/projected/6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca-kube-api-access-rm2h7\") pod \"postgres-69fb4594cf-wwb25\" (UID: \"6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:36.958174 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.958080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca-data\") pod \"postgres-69fb4594cf-wwb25\" (UID: \"6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:36.958174 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.958140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm2h7\" (UniqueName: \"kubernetes.io/projected/6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca-kube-api-access-rm2h7\") pod \"postgres-69fb4594cf-wwb25\" (UID: \"6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:36.958493 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.958475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca-data\") pod \"postgres-69fb4594cf-wwb25\" (UID: \"6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:36.966140 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:36.966113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm2h7\" (UniqueName: \"kubernetes.io/projected/6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca-kube-api-access-rm2h7\") pod \"postgres-69fb4594cf-wwb25\" (UID: \"6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:37.150669 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:37.150635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:37.282044 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:37.282019    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"]
Jul 16 20:41:37.284574 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:41:37.284546    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6d441d_0fb6_4548_beab_a8fb1ab6a0ca.slice/crio-025ac80f955aa4fde9ac4e323820b553fa933cb0b55986b70764283026c0be14 WatchSource:0}: Error finding container 025ac80f955aa4fde9ac4e323820b553fa933cb0b55986b70764283026c0be14: Status 404 returned error can't find the container with id 025ac80f955aa4fde9ac4e323820b553fa933cb0b55986b70764283026c0be14
Jul 16 20:41:37.502424 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:37.502345    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d83f14f-86cc-4dcc-bff6-8fc9e2308bda" path="/var/lib/kubelet/pods/7d83f14f-86cc-4dcc-bff6-8fc9e2308bda/volumes"
Jul 16 20:41:37.662890 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:37.662860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25" event={"ID":"6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca","Type":"ContainerStarted","Data":"025ac80f955aa4fde9ac4e323820b553fa933cb0b55986b70764283026c0be14"}
Jul 16 20:41:42.691018 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:42.690987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25" event={"ID":"6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca","Type":"ContainerStarted","Data":"331b0d993b15cfbce04b79c3080d37a8f8856dccf7ece6a74ca2f80b72924be6"}
Jul 16 20:41:42.691429 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:42.691093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:42.706322 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:42.706279    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25" podStartSLOduration=1.9442913179999999 podStartE2EDuration="6.706267105s" totalImagesPullingTime="4.761975787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:42.705547827 +0000 UTC m=+509.808826838" watchObservedRunningTime="2026-07-16 20:41:42.706267105 +0000 UTC m=+509.809546090"
Jul 16 20:41:48.724427 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:48.724397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwb25"
Jul 16 20:41:49.259745 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.259711    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-66xwc"]
Jul 16 20:41:49.262339 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.262322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-66xwc"
Jul 16 20:41:49.264930 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.264912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4pxz6\""
Jul 16 20:41:49.268734 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.268710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-66xwc"]
Jul 16 20:41:49.367475 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.367437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6xj\" (UniqueName: \"kubernetes.io/projected/e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97-kube-api-access-dv6xj\") pod \"authorino-5d4b67c446-66xwc\" (UID: \"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97\") " pod="kuadrant-system/authorino-5d4b67c446-66xwc"
Jul 16 20:41:49.468818 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.468785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6xj\" (UniqueName: \"kubernetes.io/projected/e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97-kube-api-access-dv6xj\") pod \"authorino-5d4b67c446-66xwc\" (UID: \"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97\") " pod="kuadrant-system/authorino-5d4b67c446-66xwc"
Jul 16 20:41:49.480301 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.480260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6xj\" (UniqueName: \"kubernetes.io/projected/e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97-kube-api-access-dv6xj\") pod \"authorino-5d4b67c446-66xwc\" (UID: \"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97\") " pod="kuadrant-system/authorino-5d4b67c446-66xwc"
Jul 16 20:41:49.491748 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.491717    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-66xwc"]
Jul 16 20:41:49.491962 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.491947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-66xwc"
Jul 16 20:41:49.617311 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.617283    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-66xwc"]
Jul 16 20:41:49.619543 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:41:49.619488    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2452aaa_66e3_49bd_9cd6_5b5a8b14dc97.slice/crio-43256de4074d3644524373e3bfd4281df9cefa60f0ed91860357a1480a5195ff WatchSource:0}: Error finding container 43256de4074d3644524373e3bfd4281df9cefa60f0ed91860357a1480a5195ff: Status 404 returned error can't find the container with id 43256de4074d3644524373e3bfd4281df9cefa60f0ed91860357a1480a5195ff
Jul 16 20:41:49.724594 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:49.724557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-66xwc" event={"ID":"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97","Type":"ContainerStarted","Data":"43256de4074d3644524373e3bfd4281df9cefa60f0ed91860357a1480a5195ff"}
Jul 16 20:41:52.743172 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:52.743091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-66xwc" event={"ID":"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97","Type":"ContainerStarted","Data":"5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff"}
Jul 16 20:41:52.743172 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:52.743155    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-66xwc" podUID="e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97" containerName="authorino" containerID="cri-o://5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff" gracePeriod=30
Jul 16 20:41:52.757615 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:52.757559    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-66xwc" podStartSLOduration=0.983432912 podStartE2EDuration="3.757543309s" totalImagesPullingTime="2.774110397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:52.756933227 +0000 UTC m=+519.860212432" watchObservedRunningTime="2026-07-16 20:41:52.757543309 +0000 UTC m=+519.860822294"
Jul 16 20:41:52.991808 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:52.991787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-66xwc"
Jul 16 20:41:53.105353 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.105256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv6xj\" (UniqueName: \"kubernetes.io/projected/e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97-kube-api-access-dv6xj\") pod \"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97\" (UID: \"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97\") "
Jul 16 20:41:53.107320 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.107291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97-kube-api-access-dv6xj" (OuterVolumeSpecName: "kube-api-access-dv6xj") pod "e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97" (UID: "e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97"). InnerVolumeSpecName "kube-api-access-dv6xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:41:53.206552 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.206498    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6xj\" (UniqueName: \"kubernetes.io/projected/e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97-kube-api-access-dv6xj\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:41:53.748975 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.748940    2580 generic.go:356] "Generic (PLEG): container finished" podID="e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97" containerID="5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff" exitCode=0
Jul 16 20:41:53.749411 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.748989    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-66xwc"
Jul 16 20:41:53.749411 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.749006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-66xwc" event={"ID":"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97","Type":"ContainerDied","Data":"5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff"}
Jul 16 20:41:53.749411 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.749047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-66xwc" event={"ID":"e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97","Type":"ContainerDied","Data":"43256de4074d3644524373e3bfd4281df9cefa60f0ed91860357a1480a5195ff"}
Jul 16 20:41:53.749411 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.749064    2580 scope.go:117] "RemoveContainer" containerID="5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff"
Jul 16 20:41:53.757157 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.757138    2580 scope.go:117] "RemoveContainer" containerID="5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff"
Jul 16 20:41:53.757421 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:41:53.757401    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff\": container with ID starting with 5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff not found: ID does not exist" containerID="5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff"
Jul 16 20:41:53.757488 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.757428    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff"} err="failed to get container status \"5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff\": rpc error: code = NotFound desc = could not find container \"5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff\": container with ID starting with 5d1ee258266354819dcc5e72413851cfbe190755def814d31d6a2ddfe8ecbfff not found: ID does not exist"
Jul 16 20:41:53.764506 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.764483    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-66xwc"]
Jul 16 20:41:53.768507 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:53.768488    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-66xwc"]
Jul 16 20:41:55.498111 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:55.498077    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97" path="/var/lib/kubelet/pods/e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97/volumes"
Jul 16 20:41:56.138970 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.138935    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ff65d7c9f-xjpmn"]
Jul 16 20:41:56.139241 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.139229    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97" containerName="authorino"
Jul 16 20:41:56.139283 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.139242    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97" containerName="authorino"
Jul 16 20:41:56.139316 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.139308    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2452aaa-66e3-49bd-9cd6-5b5a8b14dc97" containerName="authorino"
Jul 16 20:41:56.143851 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.143829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:56.146610 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.146590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 20:41:56.146738 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.146619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 20:41:56.146738 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.146629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hm477\""
Jul 16 20:41:56.155013 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.154987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff65d7c9f-xjpmn"]
Jul 16 20:41:56.229648 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.229612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7lt\" (UniqueName: \"kubernetes.io/projected/de1f458c-f907-4436-94f8-565e50faf683-kube-api-access-mh7lt\") pod \"maas-controller-7ff65d7c9f-xjpmn\" (UID: \"de1f458c-f907-4436-94f8-565e50faf683\") " pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:56.229830 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.229664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de1f458c-f907-4436-94f8-565e50faf683-webhook-cert\") pod \"maas-controller-7ff65d7c9f-xjpmn\" (UID: \"de1f458c-f907-4436-94f8-565e50faf683\") " pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:56.310007 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.309979    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-cvpcl"]
Jul 16 20:41:56.312352 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.312335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:56.326293 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.326270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-cvpcl"]
Jul 16 20:41:56.330779 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.330761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7lt\" (UniqueName: \"kubernetes.io/projected/de1f458c-f907-4436-94f8-565e50faf683-kube-api-access-mh7lt\") pod \"maas-controller-7ff65d7c9f-xjpmn\" (UID: \"de1f458c-f907-4436-94f8-565e50faf683\") " pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:56.330882 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.330799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de1f458c-f907-4436-94f8-565e50faf683-webhook-cert\") pod \"maas-controller-7ff65d7c9f-xjpmn\" (UID: \"de1f458c-f907-4436-94f8-565e50faf683\") " pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:56.333092 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.333065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de1f458c-f907-4436-94f8-565e50faf683-webhook-cert\") pod \"maas-controller-7ff65d7c9f-xjpmn\" (UID: \"de1f458c-f907-4436-94f8-565e50faf683\") " pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:56.341782 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.341756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7lt\" (UniqueName: \"kubernetes.io/projected/de1f458c-f907-4436-94f8-565e50faf683-kube-api-access-mh7lt\") pod \"maas-controller-7ff65d7c9f-xjpmn\" (UID: \"de1f458c-f907-4436-94f8-565e50faf683\") " pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:56.431829 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.431720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfb62666-7639-43ef-b651-7382e9b1fc13-webhook-cert\") pod \"maas-controller-5c97f6d9cb-cvpcl\" (UID: \"dfb62666-7639-43ef-b651-7382e9b1fc13\") " pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:56.431829 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.431771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnfgq\" (UniqueName: \"kubernetes.io/projected/dfb62666-7639-43ef-b651-7382e9b1fc13-kube-api-access-mnfgq\") pod \"maas-controller-5c97f6d9cb-cvpcl\" (UID: \"dfb62666-7639-43ef-b651-7382e9b1fc13\") " pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:56.453647 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.453618    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:56.533294 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.533138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfb62666-7639-43ef-b651-7382e9b1fc13-webhook-cert\") pod \"maas-controller-5c97f6d9cb-cvpcl\" (UID: \"dfb62666-7639-43ef-b651-7382e9b1fc13\") " pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:56.533294 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.533180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnfgq\" (UniqueName: \"kubernetes.io/projected/dfb62666-7639-43ef-b651-7382e9b1fc13-kube-api-access-mnfgq\") pod \"maas-controller-5c97f6d9cb-cvpcl\" (UID: \"dfb62666-7639-43ef-b651-7382e9b1fc13\") " pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:56.536047 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.535997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfb62666-7639-43ef-b651-7382e9b1fc13-webhook-cert\") pod \"maas-controller-5c97f6d9cb-cvpcl\" (UID: \"dfb62666-7639-43ef-b651-7382e9b1fc13\") " pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:56.541980 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.541942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnfgq\" (UniqueName: \"kubernetes.io/projected/dfb62666-7639-43ef-b651-7382e9b1fc13-kube-api-access-mnfgq\") pod \"maas-controller-5c97f6d9cb-cvpcl\" (UID: \"dfb62666-7639-43ef-b651-7382e9b1fc13\") " pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:56.584789 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.584764    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff65d7c9f-xjpmn"]
Jul 16 20:41:56.587009 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:41:56.586985    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1f458c_f907_4436_94f8_565e50faf683.slice/crio-fcebcc1d92b141128c83f57416412aa23bf207ff9c4776cbfad4b91660a6b88e WatchSource:0}: Error finding container fcebcc1d92b141128c83f57416412aa23bf207ff9c4776cbfad4b91660a6b88e: Status 404 returned error can't find the container with id fcebcc1d92b141128c83f57416412aa23bf207ff9c4776cbfad4b91660a6b88e
Jul 16 20:41:56.622613 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.622575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:56.765831 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.765798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn" event={"ID":"de1f458c-f907-4436-94f8-565e50faf683","Type":"ContainerStarted","Data":"fcebcc1d92b141128c83f57416412aa23bf207ff9c4776cbfad4b91660a6b88e"}
Jul 16 20:41:56.770707 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:56.770572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-cvpcl"]
Jul 16 20:41:56.773253 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:41:56.773227    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb62666_7639_43ef_b651_7382e9b1fc13.slice/crio-0963651e4908826a216fd3b98ed8d6ffbd86ddc479418207726376937a7d3e81 WatchSource:0}: Error finding container 0963651e4908826a216fd3b98ed8d6ffbd86ddc479418207726376937a7d3e81: Status 404 returned error can't find the container with id 0963651e4908826a216fd3b98ed8d6ffbd86ddc479418207726376937a7d3e81
Jul 16 20:41:57.774592 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:57.774548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl" event={"ID":"dfb62666-7639-43ef-b651-7382e9b1fc13","Type":"ContainerStarted","Data":"0963651e4908826a216fd3b98ed8d6ffbd86ddc479418207726376937a7d3e81"}
Jul 16 20:41:59.789168 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:59.789047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl" event={"ID":"dfb62666-7639-43ef-b651-7382e9b1fc13","Type":"ContainerStarted","Data":"ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1"}
Jul 16 20:41:59.789695 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:59.789165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:41:59.791262 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:59.791238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn" event={"ID":"de1f458c-f907-4436-94f8-565e50faf683","Type":"ContainerStarted","Data":"b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72"}
Jul 16 20:41:59.791461 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:59.791439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:41:59.812782 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:59.812725    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl" podStartSLOduration=1.093552729 podStartE2EDuration="3.812704391s" totalImagesPullingTime="2.719151662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:59.810928065 +0000 UTC m=+526.914207074" watchObservedRunningTime="2026-07-16 20:41:59.812704391 +0000 UTC m=+526.915983377"
Jul 16 20:41:59.830221 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:41:59.830150    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn" podStartSLOduration=1.231851032 podStartE2EDuration="3.830127388s" totalImagesPullingTime="2.598276356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:59.828360981 +0000 UTC m=+526.931639990" watchObservedRunningTime="2026-07-16 20:41:59.830127388 +0000 UTC m=+526.933406374"
Jul 16 20:42:10.797145 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:10.797066    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:42:10.797145 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:10.797141    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:42:10.859211 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:10.859179    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff65d7c9f-xjpmn"]
Jul 16 20:42:10.859417 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:10.859394    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn" podUID="de1f458c-f907-4436-94f8-565e50faf683" containerName="manager" containerID="cri-o://b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72" gracePeriod=30
Jul 16 20:42:11.108371 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.108349    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:42:11.265904 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.265870    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de1f458c-f907-4436-94f8-565e50faf683-webhook-cert\") pod \"de1f458c-f907-4436-94f8-565e50faf683\" (UID: \"de1f458c-f907-4436-94f8-565e50faf683\") "
Jul 16 20:42:11.266062 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.265962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh7lt\" (UniqueName: \"kubernetes.io/projected/de1f458c-f907-4436-94f8-565e50faf683-kube-api-access-mh7lt\") pod \"de1f458c-f907-4436-94f8-565e50faf683\" (UID: \"de1f458c-f907-4436-94f8-565e50faf683\") "
Jul 16 20:42:11.268082 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.268056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1f458c-f907-4436-94f8-565e50faf683-kube-api-access-mh7lt" (OuterVolumeSpecName: "kube-api-access-mh7lt") pod "de1f458c-f907-4436-94f8-565e50faf683" (UID: "de1f458c-f907-4436-94f8-565e50faf683"). InnerVolumeSpecName "kube-api-access-mh7lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:42:11.268191 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.268155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1f458c-f907-4436-94f8-565e50faf683-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "de1f458c-f907-4436-94f8-565e50faf683" (UID: "de1f458c-f907-4436-94f8-565e50faf683"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:42:11.367270 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.367185    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh7lt\" (UniqueName: \"kubernetes.io/projected/de1f458c-f907-4436-94f8-565e50faf683-kube-api-access-mh7lt\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:42:11.367270 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.367218    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de1f458c-f907-4436-94f8-565e50faf683-webhook-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:42:11.846054 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.846021    2580 generic.go:356] "Generic (PLEG): container finished" podID="de1f458c-f907-4436-94f8-565e50faf683" containerID="b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72" exitCode=0
Jul 16 20:42:11.846477 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.846058    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn"
Jul 16 20:42:11.846477 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.846106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn" event={"ID":"de1f458c-f907-4436-94f8-565e50faf683","Type":"ContainerDied","Data":"b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72"}
Jul 16 20:42:11.846477 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.846159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff65d7c9f-xjpmn" event={"ID":"de1f458c-f907-4436-94f8-565e50faf683","Type":"ContainerDied","Data":"fcebcc1d92b141128c83f57416412aa23bf207ff9c4776cbfad4b91660a6b88e"}
Jul 16 20:42:11.846477 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.846184    2580 scope.go:117] "RemoveContainer" containerID="b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72"
Jul 16 20:42:11.854549 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.854531    2580 scope.go:117] "RemoveContainer" containerID="b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72"
Jul 16 20:42:11.854828 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:42:11.854811    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72\": container with ID starting with b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72 not found: ID does not exist" containerID="b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72"
Jul 16 20:42:11.854875 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.854838    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72"} err="failed to get container status \"b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72\": rpc error: code = NotFound desc = could not find container \"b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72\": container with ID starting with b112086593a5a12b42fe7af37dabfc3ba76ff02bd18fda6866c539bdd0a4ab72 not found: ID does not exist"
Jul 16 20:42:11.862326 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.862304    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff65d7c9f-xjpmn"]
Jul 16 20:42:11.866200 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:11.866182    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ff65d7c9f-xjpmn"]
Jul 16 20:42:13.498135 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:13.498103    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1f458c-f907-4436-94f8-565e50faf683" path="/var/lib/kubelet/pods/de1f458c-f907-4436-94f8-565e50faf683/volumes"
Jul 16 20:42:29.307532 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.307478    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"]
Jul 16 20:42:29.308043 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.307894    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1f458c-f907-4436-94f8-565e50faf683" containerName="manager"
Jul 16 20:42:29.308043 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.307909    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1f458c-f907-4436-94f8-565e50faf683" containerName="manager"
Jul 16 20:42:29.308043 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.307967    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1f458c-f907-4436-94f8-565e50faf683" containerName="manager"
Jul 16 20:42:29.467533 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.467483    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"]
Jul 16 20:42:29.467694 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.467615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.471115 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.471089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mf4k4\""
Jul 16 20:42:29.471254 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.471125    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 16 20:42:29.499411 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.499384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e91e758-b977-4c63-a6be-6ab18bdf9f51-maas-api-tls\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.499560 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.499421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpcpt\" (UniqueName: \"kubernetes.io/projected/7e91e758-b977-4c63-a6be-6ab18bdf9f51-kube-api-access-qpcpt\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.499560 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.499446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91e758-b977-4c63-a6be-6ab18bdf9f51-openshift-service-ca\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.600882 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.600796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e91e758-b977-4c63-a6be-6ab18bdf9f51-maas-api-tls\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.600882 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.600864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpcpt\" (UniqueName: \"kubernetes.io/projected/7e91e758-b977-4c63-a6be-6ab18bdf9f51-kube-api-access-qpcpt\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.601079 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.600994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91e758-b977-4c63-a6be-6ab18bdf9f51-openshift-service-ca\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.601647 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.601626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91e758-b977-4c63-a6be-6ab18bdf9f51-openshift-service-ca\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.603171 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.603144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e91e758-b977-4c63-a6be-6ab18bdf9f51-maas-api-tls\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.613824 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.613801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpcpt\" (UniqueName: \"kubernetes.io/projected/7e91e758-b977-4c63-a6be-6ab18bdf9f51-kube-api-access-qpcpt\") pod \"maas-api-5898d5f5bf-hjllc\" (UID: \"7e91e758-b977-4c63-a6be-6ab18bdf9f51\") " pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.777711 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.777676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:29.889614 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.889584    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"]
Jul 16 20:42:29.892638 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.892616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:29.895369 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.895345    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8wh5b\""
Jul 16 20:42:29.895524 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.895496    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 16 20:42:29.902608 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.902586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxrjn\" (UniqueName: \"kubernetes.io/projected/0da53d26-42a4-4390-ac54-12a019753193-kube-api-access-kxrjn\") pod \"payload-pre-processing-7677b774bb-c9mwz\" (UID: \"0da53d26-42a4-4390-ac54-12a019753193\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:29.902726 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.902625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da53d26-42a4-4390-ac54-12a019753193-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-c9mwz\" (UID: \"0da53d26-42a4-4390-ac54-12a019753193\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:29.913115 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.913090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"]
Jul 16 20:42:29.929936 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:29.929908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"]
Jul 16 20:42:29.931807 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:42:29.931772    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e91e758_b977_4c63_a6be_6ab18bdf9f51.slice/crio-60886a2be96565d1ca20995b1dfe3fb3fdc284e82cb6ea9897dae944d9a0b464 WatchSource:0}: Error finding container 60886a2be96565d1ca20995b1dfe3fb3fdc284e82cb6ea9897dae944d9a0b464: Status 404 returned error can't find the container with id 60886a2be96565d1ca20995b1dfe3fb3fdc284e82cb6ea9897dae944d9a0b464
Jul 16 20:42:30.003462 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:30.003421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxrjn\" (UniqueName: \"kubernetes.io/projected/0da53d26-42a4-4390-ac54-12a019753193-kube-api-access-kxrjn\") pod \"payload-pre-processing-7677b774bb-c9mwz\" (UID: \"0da53d26-42a4-4390-ac54-12a019753193\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:30.003654 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:30.003477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da53d26-42a4-4390-ac54-12a019753193-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-c9mwz\" (UID: \"0da53d26-42a4-4390-ac54-12a019753193\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:30.004281 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:30.004248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da53d26-42a4-4390-ac54-12a019753193-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-c9mwz\" (UID: \"0da53d26-42a4-4390-ac54-12a019753193\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:30.018894 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:30.018874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxrjn\" (UniqueName: \"kubernetes.io/projected/0da53d26-42a4-4390-ac54-12a019753193-kube-api-access-kxrjn\") pod \"payload-pre-processing-7677b774bb-c9mwz\" (UID: \"0da53d26-42a4-4390-ac54-12a019753193\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:30.203910 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:30.203877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:30.332671 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:30.332638    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"]
Jul 16 20:42:30.336172 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:42:30.336144    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da53d26_42a4_4390_ac54_12a019753193.slice/crio-47adc4bf2747c3da07beac79916185177cf964d019a1524dc88f7b45c9386afa WatchSource:0}: Error finding container 47adc4bf2747c3da07beac79916185177cf964d019a1524dc88f7b45c9386afa: Status 404 returned error can't find the container with id 47adc4bf2747c3da07beac79916185177cf964d019a1524dc88f7b45c9386afa
Jul 16 20:42:30.933402 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:30.933362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc" event={"ID":"7e91e758-b977-4c63-a6be-6ab18bdf9f51","Type":"ContainerStarted","Data":"60886a2be96565d1ca20995b1dfe3fb3fdc284e82cb6ea9897dae944d9a0b464"}
Jul 16 20:42:30.934842 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:30.934817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz" event={"ID":"0da53d26-42a4-4390-ac54-12a019753193","Type":"ContainerStarted","Data":"47adc4bf2747c3da07beac79916185177cf964d019a1524dc88f7b45c9386afa"}
Jul 16 20:42:31.940747 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:31.940709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc" event={"ID":"7e91e758-b977-4c63-a6be-6ab18bdf9f51","Type":"ContainerStarted","Data":"89526a12b0f136cfd32bff29aaf47c4ae0803a805440138711e930c3fa13dfc9"}
Jul 16 20:42:31.941106 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:31.940892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:31.961351 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:31.961297    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc" podStartSLOduration=1.360621359 podStartE2EDuration="2.961281568s" totalImagesPullingTime="1.600660209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:42:31.960502764 +0000 UTC m=+559.063781765" watchObservedRunningTime="2026-07-16 20:42:31.961281568 +0000 UTC m=+559.064560554"
Jul 16 20:42:34.955370 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:34.955333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz" event={"ID":"0da53d26-42a4-4390-ac54-12a019753193","Type":"ContainerStarted","Data":"fd41cc4d27a363b6519632518a642c84330539f3babb16ab401f1643558b16ab"}
Jul 16 20:42:34.955861 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:34.955554    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:34.973920 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:34.973873    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz" podStartSLOduration=1.688832355 podStartE2EDuration="5.973858334s" totalImagesPullingTime="4.285025979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:42:34.973052328 +0000 UTC m=+562.076331305" watchObservedRunningTime="2026-07-16 20:42:34.973858334 +0000 UTC m=+562.077137386"
Jul 16 20:42:37.950755 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:37.950721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5898d5f5bf-hjllc"
Jul 16 20:42:45.962680 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:45.962649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-c9mwz"
Jul 16 20:42:52.582157 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.582122    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"]
Jul 16 20:42:52.585098 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.585072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.588759 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.588738    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 20:42:52.588865 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.588740    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 16 20:42:52.590150 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.590118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jqnfp\""
Jul 16 20:42:52.590242 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.590143    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 20:42:52.599836 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.599807    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"]
Jul 16 20:42:52.694099 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.694063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.694099 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.694098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.694356 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.694149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssjq\" (UniqueName: \"kubernetes.io/projected/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-kube-api-access-tssjq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.694356 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.694239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.694356 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.694297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.694356 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.694339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.795974 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.795925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.796115 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.796090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.796170 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.796141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.796214 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.796200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.796249 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.796227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.796301 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.796275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssjq\" (UniqueName: \"kubernetes.io/projected/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-kube-api-access-tssjq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.796385 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.796360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.796483 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.796462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.796683 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.796663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.798398 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.798376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.798602 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.798582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.805269 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.805248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssjq\" (UniqueName: \"kubernetes.io/projected/8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae-kube-api-access-tssjq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff\" (UID: \"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:52.898598 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:52.898563    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:42:53.030596 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:53.030567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"]
Jul 16 20:42:53.033290 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:42:53.033260    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4472e6_ee63_4b40_a6e6_ae8cd66f6cae.slice/crio-2450dd6aa98994b73e53ecc1785eb8eb990e3915f6931066c2e1eb4f990543c7 WatchSource:0}: Error finding container 2450dd6aa98994b73e53ecc1785eb8eb990e3915f6931066c2e1eb4f990543c7: Status 404 returned error can't find the container with id 2450dd6aa98994b73e53ecc1785eb8eb990e3915f6931066c2e1eb4f990543c7
Jul 16 20:42:54.039935 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:54.039898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff" event={"ID":"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae","Type":"ContainerStarted","Data":"2450dd6aa98994b73e53ecc1785eb8eb990e3915f6931066c2e1eb4f990543c7"}
Jul 16 20:42:55.046092 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:55.046058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff" event={"ID":"8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae","Type":"ContainerStarted","Data":"9eff4c555db0acc6ae12f4f117665d89d133d73d92f80c14dbac125687a5446a"}
Jul 16 20:42:55.064634 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:42:55.064579    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff" podStartSLOduration=1.928799477 podStartE2EDuration="3.064562302s" totalImagesPullingTime="1.135762825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:42:55.063841058 +0000 UTC m=+582.167120043" watchObservedRunningTime="2026-07-16 20:42:55.064562302 +0000 UTC m=+582.167841288"
Jul 16 20:43:02.899360 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:02.899329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:43:02.899960 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:02.899394    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:43:02.911637 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:02.911609    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:43:03.093314 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:03.093286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff"
Jul 16 20:43:13.330465 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:13.330439    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:43:13.331149 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:13.331125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:43:16.587201 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.587153    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"]
Jul 16 20:43:16.589538 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.589508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.592719 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.592696    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 16 20:43:16.639952 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.639921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"]
Jul 16 20:43:16.701207 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.701173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfgc\" (UniqueName: \"kubernetes.io/projected/8deb224b-0bc7-4761-aeab-5ef5b28ca136-kube-api-access-ndfgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.701362 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.701217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.701362 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.701306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8deb224b-0bc7-4761-aeab-5ef5b28ca136-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.701432 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.701370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.701432 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.701408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.701531 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.701468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.802887 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.802855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.802887 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.802890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfgc\" (UniqueName: \"kubernetes.io/projected/8deb224b-0bc7-4761-aeab-5ef5b28ca136-kube-api-access-ndfgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.803111 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.802913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.803111 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.802945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8deb224b-0bc7-4761-aeab-5ef5b28ca136-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.803111 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.802985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.803111 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.803031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.803353 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.803331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.803415 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.803401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.803461 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.803422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.805461 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.805434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8deb224b-0bc7-4761-aeab-5ef5b28ca136-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.805686 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.805670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8deb224b-0bc7-4761-aeab-5ef5b28ca136-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.812258 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.812233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfgc\" (UniqueName: \"kubernetes.io/projected/8deb224b-0bc7-4761-aeab-5ef5b28ca136-kube-api-access-ndfgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b\" (UID: \"8deb224b-0bc7-4761-aeab-5ef5b28ca136\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:16.900649 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:16.900614    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:17.042277 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:17.042255    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"]
Jul 16 20:43:17.044571 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:43:17.044542    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deb224b_0bc7_4761_aeab_5ef5b28ca136.slice/crio-cf812e451ec4eb76dfc6537982e740e213ee7e83527b2df426a41ccc354bed61 WatchSource:0}: Error finding container cf812e451ec4eb76dfc6537982e740e213ee7e83527b2df426a41ccc354bed61: Status 404 returned error can't find the container with id cf812e451ec4eb76dfc6537982e740e213ee7e83527b2df426a41ccc354bed61
Jul 16 20:43:17.143285 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:17.143251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b" event={"ID":"8deb224b-0bc7-4761-aeab-5ef5b28ca136","Type":"ContainerStarted","Data":"cf812e451ec4eb76dfc6537982e740e213ee7e83527b2df426a41ccc354bed61"}
Jul 16 20:43:18.149205 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:18.149169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b" event={"ID":"8deb224b-0bc7-4761-aeab-5ef5b28ca136","Type":"ContainerStarted","Data":"89d66a47cd8fa14a5a49eb7060b4555ddce35500e31b81c78498d64fc09b270b"}
Jul 16 20:43:18.169857 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:18.169784    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b" podStartSLOduration=1.989320784 podStartE2EDuration="2.169761745s" totalImagesPullingTime="180.440961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:43:18.169217337 +0000 UTC m=+605.272496342" watchObservedRunningTime="2026-07-16 20:43:18.169761745 +0000 UTC m=+605.273040731"
Jul 16 20:43:26.901639 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:26.901593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:26.902084 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:26.901662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:26.914354 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:26.914328    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:43:27.199745 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:43:27.199655    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b"
Jul 16 20:45:06.752635 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:06.752557    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-cvpcl"]
Jul 16 20:45:06.753040 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:06.752749    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl" podUID="dfb62666-7639-43ef-b651-7382e9b1fc13" containerName="manager" containerID="cri-o://ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1" gracePeriod=30
Jul 16 20:45:07.004352 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.004298    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:45:07.065983 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.065953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnfgq\" (UniqueName: \"kubernetes.io/projected/dfb62666-7639-43ef-b651-7382e9b1fc13-kube-api-access-mnfgq\") pod \"dfb62666-7639-43ef-b651-7382e9b1fc13\" (UID: \"dfb62666-7639-43ef-b651-7382e9b1fc13\") "
Jul 16 20:45:07.066170 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.066154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfb62666-7639-43ef-b651-7382e9b1fc13-webhook-cert\") pod \"dfb62666-7639-43ef-b651-7382e9b1fc13\" (UID: \"dfb62666-7639-43ef-b651-7382e9b1fc13\") "
Jul 16 20:45:07.068148 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.068119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb62666-7639-43ef-b651-7382e9b1fc13-kube-api-access-mnfgq" (OuterVolumeSpecName: "kube-api-access-mnfgq") pod "dfb62666-7639-43ef-b651-7382e9b1fc13" (UID: "dfb62666-7639-43ef-b651-7382e9b1fc13"). InnerVolumeSpecName "kube-api-access-mnfgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:45:07.068269 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.068241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb62666-7639-43ef-b651-7382e9b1fc13-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dfb62666-7639-43ef-b651-7382e9b1fc13" (UID: "dfb62666-7639-43ef-b651-7382e9b1fc13"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:45:07.167118 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.167086    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfb62666-7639-43ef-b651-7382e9b1fc13-webhook-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:45:07.167118 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.167113    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnfgq\" (UniqueName: \"kubernetes.io/projected/dfb62666-7639-43ef-b651-7382e9b1fc13-kube-api-access-mnfgq\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:45:07.638477 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.638440    2580 generic.go:356] "Generic (PLEG): container finished" podID="dfb62666-7639-43ef-b651-7382e9b1fc13" containerID="ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1" exitCode=0
Jul 16 20:45:07.638477 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.638478    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl"
Jul 16 20:45:07.638711 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.638562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl" event={"ID":"dfb62666-7639-43ef-b651-7382e9b1fc13","Type":"ContainerDied","Data":"ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1"}
Jul 16 20:45:07.638711 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.638614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-cvpcl" event={"ID":"dfb62666-7639-43ef-b651-7382e9b1fc13","Type":"ContainerDied","Data":"0963651e4908826a216fd3b98ed8d6ffbd86ddc479418207726376937a7d3e81"}
Jul 16 20:45:07.638711 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.638635    2580 scope.go:117] "RemoveContainer" containerID="ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1"
Jul 16 20:45:07.646777 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.646758    2580 scope.go:117] "RemoveContainer" containerID="ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1"
Jul 16 20:45:07.647028 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:45:07.647007    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1\": container with ID starting with ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1 not found: ID does not exist" containerID="ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1"
Jul 16 20:45:07.647075 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.647040    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1"} err="failed to get container status \"ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1\": rpc error: code = NotFound desc = could not find container \"ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1\": container with ID starting with ae5eaef0b62a2f26e845496c4eea9c48f1d88d26f605cbcccb46acb7cc53ced1 not found: ID does not exist"
Jul 16 20:45:07.656643 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.656622    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-cvpcl"]
Jul 16 20:45:07.663349 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:07.663329    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-cvpcl"]
Jul 16 20:45:09.498646 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:45:09.498615    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfb62666-7639-43ef-b651-7382e9b1fc13" path="/var/lib/kubelet/pods/dfb62666-7639-43ef-b651-7382e9b1fc13/volumes"
Jul 16 20:46:08.490306 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.490261    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-vzkg4"]
Jul 16 20:46:08.490810 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.490657    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb62666-7639-43ef-b651-7382e9b1fc13" containerName="manager"
Jul 16 20:46:08.490810 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.490671    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb62666-7639-43ef-b651-7382e9b1fc13" containerName="manager"
Jul 16 20:46:08.490810 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.490749    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb62666-7639-43ef-b651-7382e9b1fc13" containerName="manager"
Jul 16 20:46:08.492773 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.492753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:08.496470 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.496451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 20:46:08.496614 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.496451    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hm477\""
Jul 16 20:46:08.496614 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.496451    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 20:46:08.502173 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.502144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-vzkg4"]
Jul 16 20:46:08.616502 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.616456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmng\" (UniqueName: \"kubernetes.io/projected/b043acee-abb6-4dbc-8cc7-670fc821f326-kube-api-access-ltmng\") pod \"maas-controller-5c97f6d9cb-vzkg4\" (UID: \"b043acee-abb6-4dbc-8cc7-670fc821f326\") " pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:08.616695 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.616602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b043acee-abb6-4dbc-8cc7-670fc821f326-webhook-cert\") pod \"maas-controller-5c97f6d9cb-vzkg4\" (UID: \"b043acee-abb6-4dbc-8cc7-670fc821f326\") " pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:08.717725 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.717696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b043acee-abb6-4dbc-8cc7-670fc821f326-webhook-cert\") pod \"maas-controller-5c97f6d9cb-vzkg4\" (UID: \"b043acee-abb6-4dbc-8cc7-670fc821f326\") " pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:08.717887 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.717781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmng\" (UniqueName: \"kubernetes.io/projected/b043acee-abb6-4dbc-8cc7-670fc821f326-kube-api-access-ltmng\") pod \"maas-controller-5c97f6d9cb-vzkg4\" (UID: \"b043acee-abb6-4dbc-8cc7-670fc821f326\") " pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:08.720050 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.720024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b043acee-abb6-4dbc-8cc7-670fc821f326-webhook-cert\") pod \"maas-controller-5c97f6d9cb-vzkg4\" (UID: \"b043acee-abb6-4dbc-8cc7-670fc821f326\") " pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:08.726730 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.726708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmng\" (UniqueName: \"kubernetes.io/projected/b043acee-abb6-4dbc-8cc7-670fc821f326-kube-api-access-ltmng\") pod \"maas-controller-5c97f6d9cb-vzkg4\" (UID: \"b043acee-abb6-4dbc-8cc7-670fc821f326\") " pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:08.805103 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.805021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:08.940978 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.940955    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-vzkg4"]
Jul 16 20:46:08.943656 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:46:08.943627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb043acee_abb6_4dbc_8cc7_670fc821f326.slice/crio-d87aa2de7ed634487735904ac66da3be6ad1631376df1e08b063670995a783ae WatchSource:0}: Error finding container d87aa2de7ed634487735904ac66da3be6ad1631376df1e08b063670995a783ae: Status 404 returned error can't find the container with id d87aa2de7ed634487735904ac66da3be6ad1631376df1e08b063670995a783ae
Jul 16 20:46:08.945033 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:08.945012    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:46:09.918735 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:09.918701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4" event={"ID":"b043acee-abb6-4dbc-8cc7-670fc821f326","Type":"ContainerStarted","Data":"c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828"}
Jul 16 20:46:09.919120 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:09.918757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4" event={"ID":"b043acee-abb6-4dbc-8cc7-670fc821f326","Type":"ContainerStarted","Data":"d87aa2de7ed634487735904ac66da3be6ad1631376df1e08b063670995a783ae"}
Jul 16 20:46:09.919120 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:09.918791    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:09.935399 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:09.935352    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4" podStartSLOduration=1.406567207 podStartE2EDuration="1.935339063s" totalImagesPullingTime="528.771856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:46:09.933864078 +0000 UTC m=+777.037143088" watchObservedRunningTime="2026-07-16 20:46:09.935339063 +0000 UTC m=+777.038618078"
Jul 16 20:46:20.926916 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:20.926882    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:25.547531 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.547477    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-vzkg4"]
Jul 16 20:46:25.548033 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.547699    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4" podUID="b043acee-abb6-4dbc-8cc7-670fc821f326" containerName="manager" containerID="cri-o://c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828" gracePeriod=30
Jul 16 20:46:25.788991 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.788962    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:25.975392 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.975358    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b043acee-abb6-4dbc-8cc7-670fc821f326-webhook-cert\") pod \"b043acee-abb6-4dbc-8cc7-670fc821f326\" (UID: \"b043acee-abb6-4dbc-8cc7-670fc821f326\") "
Jul 16 20:46:25.975578 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.975420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltmng\" (UniqueName: \"kubernetes.io/projected/b043acee-abb6-4dbc-8cc7-670fc821f326-kube-api-access-ltmng\") pod \"b043acee-abb6-4dbc-8cc7-670fc821f326\" (UID: \"b043acee-abb6-4dbc-8cc7-670fc821f326\") "
Jul 16 20:46:25.977486 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.977456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b043acee-abb6-4dbc-8cc7-670fc821f326-kube-api-access-ltmng" (OuterVolumeSpecName: "kube-api-access-ltmng") pod "b043acee-abb6-4dbc-8cc7-670fc821f326" (UID: "b043acee-abb6-4dbc-8cc7-670fc821f326"). InnerVolumeSpecName "kube-api-access-ltmng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:46:25.977619 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.977597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b043acee-abb6-4dbc-8cc7-670fc821f326-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b043acee-abb6-4dbc-8cc7-670fc821f326" (UID: "b043acee-abb6-4dbc-8cc7-670fc821f326"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:46:25.992697 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.992670    2580 generic.go:356] "Generic (PLEG): container finished" podID="b043acee-abb6-4dbc-8cc7-670fc821f326" containerID="c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828" exitCode=0
Jul 16 20:46:25.992807 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.992716    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4"
Jul 16 20:46:25.992807 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.992759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4" event={"ID":"b043acee-abb6-4dbc-8cc7-670fc821f326","Type":"ContainerDied","Data":"c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828"}
Jul 16 20:46:25.992917 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.992819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-vzkg4" event={"ID":"b043acee-abb6-4dbc-8cc7-670fc821f326","Type":"ContainerDied","Data":"d87aa2de7ed634487735904ac66da3be6ad1631376df1e08b063670995a783ae"}
Jul 16 20:46:25.992917 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:25.992846    2580 scope.go:117] "RemoveContainer" containerID="c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828"
Jul 16 20:46:26.001155 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.001140    2580 scope.go:117] "RemoveContainer" containerID="c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828"
Jul 16 20:46:26.001415 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:46:26.001386    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828\": container with ID starting with c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828 not found: ID does not exist" containerID="c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828"
Jul 16 20:46:26.001465 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.001420    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828"} err="failed to get container status \"c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828\": rpc error: code = NotFound desc = could not find container \"c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828\": container with ID starting with c89048a96954d83db315411737dc95df132e51fd1dd28f910342b695b64d1828 not found: ID does not exist"
Jul 16 20:46:26.014134 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.014112    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-vzkg4"]
Jul 16 20:46:26.018620 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.018599    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-vzkg4"]
Jul 16 20:46:26.076650 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.076621    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b043acee-abb6-4dbc-8cc7-670fc821f326-webhook-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:46:26.076650 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.076648    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmng\" (UniqueName: \"kubernetes.io/projected/b043acee-abb6-4dbc-8cc7-670fc821f326-kube-api-access-ltmng\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:46:26.799030 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.798994    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-c9ngr"]
Jul 16 20:46:26.799378 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.799350    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b043acee-abb6-4dbc-8cc7-670fc821f326" containerName="manager"
Jul 16 20:46:26.799378 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.799361    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b043acee-abb6-4dbc-8cc7-670fc821f326" containerName="manager"
Jul 16 20:46:26.799451 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.799445    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b043acee-abb6-4dbc-8cc7-670fc821f326" containerName="manager"
Jul 16 20:46:26.802296 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.802278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:26.806070 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.806032    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 20:46:26.806070 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.806045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hm477\""
Jul 16 20:46:26.806244 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.806073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 20:46:26.811804 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.811780    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-c9ngr"]
Jul 16 20:46:26.984064 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.984033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff7dd01c-c58b-4744-a59b-7f26b578e0ae-webhook-cert\") pod \"maas-controller-5c97f6d9cb-c9ngr\" (UID: \"ff7dd01c-c58b-4744-a59b-7f26b578e0ae\") " pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:26.984220 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:26.984092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfzl\" (UniqueName: \"kubernetes.io/projected/ff7dd01c-c58b-4744-a59b-7f26b578e0ae-kube-api-access-xsfzl\") pod \"maas-controller-5c97f6d9cb-c9ngr\" (UID: \"ff7dd01c-c58b-4744-a59b-7f26b578e0ae\") " pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:27.084745 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:27.084656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfzl\" (UniqueName: \"kubernetes.io/projected/ff7dd01c-c58b-4744-a59b-7f26b578e0ae-kube-api-access-xsfzl\") pod \"maas-controller-5c97f6d9cb-c9ngr\" (UID: \"ff7dd01c-c58b-4744-a59b-7f26b578e0ae\") " pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:27.084745 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:27.084742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff7dd01c-c58b-4744-a59b-7f26b578e0ae-webhook-cert\") pod \"maas-controller-5c97f6d9cb-c9ngr\" (UID: \"ff7dd01c-c58b-4744-a59b-7f26b578e0ae\") " pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:27.087142 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:27.087124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff7dd01c-c58b-4744-a59b-7f26b578e0ae-webhook-cert\") pod \"maas-controller-5c97f6d9cb-c9ngr\" (UID: \"ff7dd01c-c58b-4744-a59b-7f26b578e0ae\") " pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:27.094288 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:27.094262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfzl\" (UniqueName: \"kubernetes.io/projected/ff7dd01c-c58b-4744-a59b-7f26b578e0ae-kube-api-access-xsfzl\") pod \"maas-controller-5c97f6d9cb-c9ngr\" (UID: \"ff7dd01c-c58b-4744-a59b-7f26b578e0ae\") " pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:27.112925 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:27.112902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:27.245739 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:27.245712    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c97f6d9cb-c9ngr"]
Jul 16 20:46:27.248083 ip-10-0-140-200 kubenswrapper[2580]: W0716 20:46:27.248047    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7dd01c_c58b_4744_a59b_7f26b578e0ae.slice/crio-351d00378734645af16dcf72a04cbbd6cbaadb185f858678ae250867f6ea04d7 WatchSource:0}: Error finding container 351d00378734645af16dcf72a04cbbd6cbaadb185f858678ae250867f6ea04d7: Status 404 returned error can't find the container with id 351d00378734645af16dcf72a04cbbd6cbaadb185f858678ae250867f6ea04d7
Jul 16 20:46:27.498656 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:27.498623    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b043acee-abb6-4dbc-8cc7-670fc821f326" path="/var/lib/kubelet/pods/b043acee-abb6-4dbc-8cc7-670fc821f326/volumes"
Jul 16 20:46:28.003260 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:28.003222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr" event={"ID":"ff7dd01c-c58b-4744-a59b-7f26b578e0ae","Type":"ContainerStarted","Data":"cac99818d0ffa82be42373e4edaf69b23c124c34eef9c88c6a9e0ca1144dbb45"}
Jul 16 20:46:28.003670 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:28.003310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr" event={"ID":"ff7dd01c-c58b-4744-a59b-7f26b578e0ae","Type":"ContainerStarted","Data":"351d00378734645af16dcf72a04cbbd6cbaadb185f858678ae250867f6ea04d7"}
Jul 16 20:46:28.003670 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:28.003337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:46:28.022221 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:28.022121    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr" podStartSLOduration=1.506387801 podStartE2EDuration="2.022103744s" totalImagesPullingTime="515.715943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:46:28.020966091 +0000 UTC m=+795.124245075" watchObservedRunningTime="2026-07-16 20:46:28.022103744 +0000 UTC m=+795.125382730"
Jul 16 20:46:39.010485 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:46:39.010410    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c97f6d9cb-c9ngr"
Jul 16 20:48:13.360285 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:48:13.360198    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:48:13.360770 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:48:13.360316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:53:13.393963 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:53:13.393936    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:53:13.394821 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:53:13.394806    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:58:13.423630 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:13.423595    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:58:13.424160 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:13.424108    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 20:58:47.467085 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.467049    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"]
Jul 16 20:58:47.470144 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.470121    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.473473 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.473452    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-866d7766-serving-cert\""
Jul 16 20:58:47.482225 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.482200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"]
Jul 16 20:58:47.518600 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.518562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66d7aa55-8b62-4f6a-8844-b66ac24ae705-maas-api-tls\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.519009 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.518663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d7aa55-8b62-4f6a-8844-b66ac24ae705-openshift-service-ca\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.519009 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.518716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkhs\" (UniqueName: \"kubernetes.io/projected/66d7aa55-8b62-4f6a-8844-b66ac24ae705-kube-api-access-ldkhs\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.619381 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.619343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkhs\" (UniqueName: \"kubernetes.io/projected/66d7aa55-8b62-4f6a-8844-b66ac24ae705-kube-api-access-ldkhs\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.619612 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.619416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66d7aa55-8b62-4f6a-8844-b66ac24ae705-maas-api-tls\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.619612 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.619485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d7aa55-8b62-4f6a-8844-b66ac24ae705-openshift-service-ca\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.620291 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.620262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d7aa55-8b62-4f6a-8844-b66ac24ae705-openshift-service-ca\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.622020 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.622000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66d7aa55-8b62-4f6a-8844-b66ac24ae705-maas-api-tls\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.627698 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.627665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkhs\" (UniqueName: \"kubernetes.io/projected/66d7aa55-8b62-4f6a-8844-b66ac24ae705-kube-api-access-ldkhs\") pod \"maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.780749 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.780656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:47.922729 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.917891    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"]
Jul 16 20:58:47.923695 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:47.923660    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:58:48.319764 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:48.319728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9" event={"ID":"66d7aa55-8b62-4f6a-8844-b66ac24ae705","Type":"ContainerStarted","Data":"af2d014bd55dbb8f9b2eec9700c62524fb7d68e992e641bf5bce1b2ecbf2e82d"}
Jul 16 20:58:49.325670 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:49.325630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9" event={"ID":"66d7aa55-8b62-4f6a-8844-b66ac24ae705","Type":"ContainerStarted","Data":"48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781"}
Jul 16 20:58:49.326061 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:49.325749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:58:49.345472 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:49.345418    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9" podStartSLOduration=1.8158556510000001 podStartE2EDuration="2.345401078s" totalImagesPullingTime="529.545427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:58:49.343834473 +0000 UTC m=+1536.447113471" watchObservedRunningTime="2026-07-16 20:58:49.345401078 +0000 UTC m=+1536.448680063"
Jul 16 20:58:55.335387 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:58:55.335358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:59:52.499657 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.499617    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"]
Jul 16 20:59:52.502223 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.499851    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9" podUID="66d7aa55-8b62-4f6a-8844-b66ac24ae705" containerName="maas-api" containerID="cri-o://48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781" gracePeriod=30
Jul 16 20:59:52.746714 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.746689    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:59:52.795681 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.795597    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d7aa55-8b62-4f6a-8844-b66ac24ae705-openshift-service-ca\") pod \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") "
Jul 16 20:59:52.795681 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.795665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldkhs\" (UniqueName: \"kubernetes.io/projected/66d7aa55-8b62-4f6a-8844-b66ac24ae705-kube-api-access-ldkhs\") pod \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") "
Jul 16 20:59:52.795871 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.795773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66d7aa55-8b62-4f6a-8844-b66ac24ae705-maas-api-tls\") pod \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\" (UID: \"66d7aa55-8b62-4f6a-8844-b66ac24ae705\") "
Jul 16 20:59:52.796034 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.796015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d7aa55-8b62-4f6a-8844-b66ac24ae705-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66d7aa55-8b62-4f6a-8844-b66ac24ae705" (UID: "66d7aa55-8b62-4f6a-8844-b66ac24ae705"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:59:52.798005 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.797979    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d7aa55-8b62-4f6a-8844-b66ac24ae705-kube-api-access-ldkhs" (OuterVolumeSpecName: "kube-api-access-ldkhs") pod "66d7aa55-8b62-4f6a-8844-b66ac24ae705" (UID: "66d7aa55-8b62-4f6a-8844-b66ac24ae705"). InnerVolumeSpecName "kube-api-access-ldkhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:59:52.798005 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.797990    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d7aa55-8b62-4f6a-8844-b66ac24ae705-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66d7aa55-8b62-4f6a-8844-b66ac24ae705" (UID: "66d7aa55-8b62-4f6a-8844-b66ac24ae705"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:59:52.896662 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.896627    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d7aa55-8b62-4f6a-8844-b66ac24ae705-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:59:52.896662 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.896659    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldkhs\" (UniqueName: \"kubernetes.io/projected/66d7aa55-8b62-4f6a-8844-b66ac24ae705-kube-api-access-ldkhs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:59:52.896662 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:52.896669    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66d7aa55-8b62-4f6a-8844-b66ac24ae705-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 20:59:53.612356 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.612319    2580 generic.go:356] "Generic (PLEG): container finished" podID="66d7aa55-8b62-4f6a-8844-b66ac24ae705" containerID="48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781" exitCode=0
Jul 16 20:59:53.612849 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.612365    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"
Jul 16 20:59:53.612849 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.612403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9" event={"ID":"66d7aa55-8b62-4f6a-8844-b66ac24ae705","Type":"ContainerDied","Data":"48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781"}
Jul 16 20:59:53.612849 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.612456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9" event={"ID":"66d7aa55-8b62-4f6a-8844-b66ac24ae705","Type":"ContainerDied","Data":"af2d014bd55dbb8f9b2eec9700c62524fb7d68e992e641bf5bce1b2ecbf2e82d"}
Jul 16 20:59:53.612849 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.612483    2580 scope.go:117] "RemoveContainer" containerID="48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781"
Jul 16 20:59:53.620747 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.620731    2580 scope.go:117] "RemoveContainer" containerID="48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781"
Jul 16 20:59:53.621020 ip-10-0-140-200 kubenswrapper[2580]: E0716 20:59:53.621001    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781\": container with ID starting with 48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781 not found: ID does not exist" containerID="48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781"
Jul 16 20:59:53.621065 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.621032    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781"} err="failed to get container status \"48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781\": rpc error: code = NotFound desc = could not find container \"48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781\": container with ID starting with 48d97a7fb239f8fced914ba56037e0f62232a72ae5f292ae4cc822046f611781 not found: ID does not exist"
Jul 16 20:59:53.628962 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.628937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"]
Jul 16 20:59:53.631256 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:53.631233    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-866d7766-55dcd8868d-qwqp9"]
Jul 16 20:59:55.498943 ip-10-0-140-200 kubenswrapper[2580]: I0716 20:59:55.498907    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d7aa55-8b62-4f6a-8844-b66ac24ae705" path="/var/lib/kubelet/pods/66d7aa55-8b62-4f6a-8844-b66ac24ae705/volumes"
Jul 16 21:00:05.382650 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.382616    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"]
Jul 16 21:00:05.383031 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.382959    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d7aa55-8b62-4f6a-8844-b66ac24ae705" containerName="maas-api"
Jul 16 21:00:05.383031 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.382969    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d7aa55-8b62-4f6a-8844-b66ac24ae705" containerName="maas-api"
Jul 16 21:00:05.383110 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.383037    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d7aa55-8b62-4f6a-8844-b66ac24ae705" containerName="maas-api"
Jul 16 21:00:05.387437 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.387412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.390032 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.390006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-18adf534-serving-cert\""
Jul 16 21:00:05.396043 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.396015    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"]
Jul 16 21:00:05.504407 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.504370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94d1e8a-100f-4379-a6ae-777dca0f86e8-maas-api-tls\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.504603 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.504425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94d1e8a-100f-4379-a6ae-777dca0f86e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.504603 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.504552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdtpq\" (UniqueName: \"kubernetes.io/projected/b94d1e8a-100f-4379-a6ae-777dca0f86e8-kube-api-access-mdtpq\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.605136 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.605106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94d1e8a-100f-4379-a6ae-777dca0f86e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.605285 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.605164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdtpq\" (UniqueName: \"kubernetes.io/projected/b94d1e8a-100f-4379-a6ae-777dca0f86e8-kube-api-access-mdtpq\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.605285 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.605194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94d1e8a-100f-4379-a6ae-777dca0f86e8-maas-api-tls\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.605792 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.605770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94d1e8a-100f-4379-a6ae-777dca0f86e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.607655 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.607634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94d1e8a-100f-4379-a6ae-777dca0f86e8-maas-api-tls\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.614128 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.614101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdtpq\" (UniqueName: \"kubernetes.io/projected/b94d1e8a-100f-4379-a6ae-777dca0f86e8-kube-api-access-mdtpq\") pod \"maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.700084 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.700045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:05.846345 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:05.846289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"]
Jul 16 21:00:05.848699 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:00:05.848671    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94d1e8a_100f_4379_a6ae_777dca0f86e8.slice/crio-a39a523b170c59e6aac14dd51d17d1991364502cbe2b8f0ce04dc82b7d2a60ea WatchSource:0}: Error finding container a39a523b170c59e6aac14dd51d17d1991364502cbe2b8f0ce04dc82b7d2a60ea: Status 404 returned error can't find the container with id a39a523b170c59e6aac14dd51d17d1991364502cbe2b8f0ce04dc82b7d2a60ea
Jul 16 21:00:06.674308 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:06.674256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626" event={"ID":"b94d1e8a-100f-4379-a6ae-777dca0f86e8","Type":"ContainerStarted","Data":"32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8"}
Jul 16 21:00:06.674701 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:06.674328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626" event={"ID":"b94d1e8a-100f-4379-a6ae-777dca0f86e8","Type":"ContainerStarted","Data":"a39a523b170c59e6aac14dd51d17d1991364502cbe2b8f0ce04dc82b7d2a60ea"}
Jul 16 21:00:06.674701 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:06.674356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:00:06.692104 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:06.692056    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626" podStartSLOduration=1.239444481 podStartE2EDuration="1.69203904s" totalImagesPullingTime="452.594559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:00:06.691011758 +0000 UTC m=+1613.794290743" watchObservedRunningTime="2026-07-16 21:00:06.69203904 +0000 UTC m=+1613.795318027"
Jul 16 21:00:12.683874 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:00:12.683846    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:01:09.921620 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:09.921585    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"]
Jul 16 21:01:09.922122 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:09.921845    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626" podUID="b94d1e8a-100f-4379-a6ae-777dca0f86e8" containerName="maas-api" containerID="cri-o://32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8" gracePeriod=30
Jul 16 21:01:10.173351 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.173287    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:01:10.283781 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.283742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdtpq\" (UniqueName: \"kubernetes.io/projected/b94d1e8a-100f-4379-a6ae-777dca0f86e8-kube-api-access-mdtpq\") pod \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") "
Jul 16 21:01:10.283781 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.283789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94d1e8a-100f-4379-a6ae-777dca0f86e8-maas-api-tls\") pod \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") "
Jul 16 21:01:10.284026 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.283854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94d1e8a-100f-4379-a6ae-777dca0f86e8-openshift-service-ca\") pod \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\" (UID: \"b94d1e8a-100f-4379-a6ae-777dca0f86e8\") "
Jul 16 21:01:10.284314 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.284286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94d1e8a-100f-4379-a6ae-777dca0f86e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b94d1e8a-100f-4379-a6ae-777dca0f86e8" (UID: "b94d1e8a-100f-4379-a6ae-777dca0f86e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:01:10.285982 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.285956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94d1e8a-100f-4379-a6ae-777dca0f86e8-kube-api-access-mdtpq" (OuterVolumeSpecName: "kube-api-access-mdtpq") pod "b94d1e8a-100f-4379-a6ae-777dca0f86e8" (UID: "b94d1e8a-100f-4379-a6ae-777dca0f86e8"). InnerVolumeSpecName "kube-api-access-mdtpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:01:10.285982 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.285966    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94d1e8a-100f-4379-a6ae-777dca0f86e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b94d1e8a-100f-4379-a6ae-777dca0f86e8" (UID: "b94d1e8a-100f-4379-a6ae-777dca0f86e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:01:10.385261 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.385220    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdtpq\" (UniqueName: \"kubernetes.io/projected/b94d1e8a-100f-4379-a6ae-777dca0f86e8-kube-api-access-mdtpq\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:01:10.385261 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.385258    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94d1e8a-100f-4379-a6ae-777dca0f86e8-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:01:10.385468 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.385271    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94d1e8a-100f-4379-a6ae-777dca0f86e8-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:01:10.964199 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.964165    2580 generic.go:356] "Generic (PLEG): container finished" podID="b94d1e8a-100f-4379-a6ae-777dca0f86e8" containerID="32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8" exitCode=0
Jul 16 21:01:10.964624 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.964206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626" event={"ID":"b94d1e8a-100f-4379-a6ae-777dca0f86e8","Type":"ContainerDied","Data":"32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8"}
Jul 16 21:01:10.964624 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.964215    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"
Jul 16 21:01:10.964624 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.964251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626" event={"ID":"b94d1e8a-100f-4379-a6ae-777dca0f86e8","Type":"ContainerDied","Data":"a39a523b170c59e6aac14dd51d17d1991364502cbe2b8f0ce04dc82b7d2a60ea"}
Jul 16 21:01:10.964624 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.964283    2580 scope.go:117] "RemoveContainer" containerID="32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8"
Jul 16 21:01:10.974359 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.974333    2580 scope.go:117] "RemoveContainer" containerID="32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8"
Jul 16 21:01:10.974667 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:01:10.974647    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8\": container with ID starting with 32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8 not found: ID does not exist" containerID="32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8"
Jul 16 21:01:10.974733 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.974675    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8"} err="failed to get container status \"32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8\": rpc error: code = NotFound desc = could not find container \"32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8\": container with ID starting with 32878f66589618b8f3b5d2e55f2dbc57a0d28fd46ca84a191c4927ca8058f4f8 not found: ID does not exist"
Jul 16 21:01:10.985946 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.985915    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"]
Jul 16 21:01:10.988351 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:10.988326    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18adf534-7bd44ff7f-v5626"]
Jul 16 21:01:11.498895 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:11.498860    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94d1e8a-100f-4379-a6ae-777dca0f86e8" path="/var/lib/kubelet/pods/b94d1e8a-100f-4379-a6ae-777dca0f86e8/volumes"
Jul 16 21:01:17.469742 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.469706    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"]
Jul 16 21:01:17.470113 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.470063    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94d1e8a-100f-4379-a6ae-777dca0f86e8" containerName="maas-api"
Jul 16 21:01:17.470113 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.470073    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94d1e8a-100f-4379-a6ae-777dca0f86e8" containerName="maas-api"
Jul 16 21:01:17.470181 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.470123    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94d1e8a-100f-4379-a6ae-777dca0f86e8" containerName="maas-api"
Jul 16 21:01:17.473120 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.473103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.476163 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.476142    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-25df27a3-serving-cert\""
Jul 16 21:01:17.500318 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.500281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"]
Jul 16 21:01:17.648005 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.647953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nnx8\" (UniqueName: \"kubernetes.io/projected/41e8f28e-ee8c-4769-9898-e19348d873cd-kube-api-access-7nnx8\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.648205 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.648033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e8f28e-ee8c-4769-9898-e19348d873cd-openshift-service-ca\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.648205 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.648087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e8f28e-ee8c-4769-9898-e19348d873cd-maas-api-tls\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.748674 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.748576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nnx8\" (UniqueName: \"kubernetes.io/projected/41e8f28e-ee8c-4769-9898-e19348d873cd-kube-api-access-7nnx8\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.748674 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.748634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e8f28e-ee8c-4769-9898-e19348d873cd-openshift-service-ca\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.748896 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.748681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e8f28e-ee8c-4769-9898-e19348d873cd-maas-api-tls\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.749877 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.749830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e8f28e-ee8c-4769-9898-e19348d873cd-openshift-service-ca\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.751826 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.751797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e8f28e-ee8c-4769-9898-e19348d873cd-maas-api-tls\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.758546 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.758507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nnx8\" (UniqueName: \"kubernetes.io/projected/41e8f28e-ee8c-4769-9898-e19348d873cd-kube-api-access-7nnx8\") pod \"maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.782741 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.782709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:17.930215 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:17.930183    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"]
Jul 16 21:01:17.932576 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:01:17.932544    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e8f28e_ee8c_4769_9898_e19348d873cd.slice/crio-675b5bb36aafe0fbd911b8653346a1d4d3d94a09294f17a02eddef180b07a2f6 WatchSource:0}: Error finding container 675b5bb36aafe0fbd911b8653346a1d4d3d94a09294f17a02eddef180b07a2f6: Status 404 returned error can't find the container with id 675b5bb36aafe0fbd911b8653346a1d4d3d94a09294f17a02eddef180b07a2f6
Jul 16 21:01:18.000823 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:18.000741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b" event={"ID":"41e8f28e-ee8c-4769-9898-e19348d873cd","Type":"ContainerStarted","Data":"675b5bb36aafe0fbd911b8653346a1d4d3d94a09294f17a02eddef180b07a2f6"}
Jul 16 21:01:19.007357 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:19.007317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b" event={"ID":"41e8f28e-ee8c-4769-9898-e19348d873cd","Type":"ContainerStarted","Data":"62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc"}
Jul 16 21:01:19.007864 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:19.007423    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:01:19.025671 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:19.025612    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b" podStartSLOduration=1.542711043 podStartE2EDuration="2.025595318s" totalImagesPullingTime="482.884275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:01:19.02471443 +0000 UTC m=+1686.127993434" watchObservedRunningTime="2026-07-16 21:01:19.025595318 +0000 UTC m=+1686.128874304"
Jul 16 21:01:25.017122 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:01:25.017087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:02:32.653993 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:32.653937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"]
Jul 16 21:02:32.654573 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:32.654171    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b" podUID="41e8f28e-ee8c-4769-9898-e19348d873cd" containerName="maas-api" containerID="cri-o://62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc" gracePeriod=30
Jul 16 21:02:32.902828 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:32.902801    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:02:33.004413 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.004326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e8f28e-ee8c-4769-9898-e19348d873cd-openshift-service-ca\") pod \"41e8f28e-ee8c-4769-9898-e19348d873cd\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") "
Jul 16 21:02:33.004413 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.004379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e8f28e-ee8c-4769-9898-e19348d873cd-maas-api-tls\") pod \"41e8f28e-ee8c-4769-9898-e19348d873cd\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") "
Jul 16 21:02:33.004683 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.004434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nnx8\" (UniqueName: \"kubernetes.io/projected/41e8f28e-ee8c-4769-9898-e19348d873cd-kube-api-access-7nnx8\") pod \"41e8f28e-ee8c-4769-9898-e19348d873cd\" (UID: \"41e8f28e-ee8c-4769-9898-e19348d873cd\") "
Jul 16 21:02:33.004764 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.004734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e8f28e-ee8c-4769-9898-e19348d873cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41e8f28e-ee8c-4769-9898-e19348d873cd" (UID: "41e8f28e-ee8c-4769-9898-e19348d873cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:02:33.006549 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.006522    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e8f28e-ee8c-4769-9898-e19348d873cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41e8f28e-ee8c-4769-9898-e19348d873cd" (UID: "41e8f28e-ee8c-4769-9898-e19348d873cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:02:33.006664 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.006549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e8f28e-ee8c-4769-9898-e19348d873cd-kube-api-access-7nnx8" (OuterVolumeSpecName: "kube-api-access-7nnx8") pod "41e8f28e-ee8c-4769-9898-e19348d873cd" (UID: "41e8f28e-ee8c-4769-9898-e19348d873cd"). InnerVolumeSpecName "kube-api-access-7nnx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:02:33.105918 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.105864    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nnx8\" (UniqueName: \"kubernetes.io/projected/41e8f28e-ee8c-4769-9898-e19348d873cd-kube-api-access-7nnx8\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:02:33.105918 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.105910    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e8f28e-ee8c-4769-9898-e19348d873cd-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:02:33.105918 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.105921    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e8f28e-ee8c-4769-9898-e19348d873cd-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:02:33.348910 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.348816    2580 generic.go:356] "Generic (PLEG): container finished" podID="41e8f28e-ee8c-4769-9898-e19348d873cd" containerID="62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc" exitCode=0
Jul 16 21:02:33.348910 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.348864    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"
Jul 16 21:02:33.349107 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.348865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b" event={"ID":"41e8f28e-ee8c-4769-9898-e19348d873cd","Type":"ContainerDied","Data":"62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc"}
Jul 16 21:02:33.349107 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.349014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b" event={"ID":"41e8f28e-ee8c-4769-9898-e19348d873cd","Type":"ContainerDied","Data":"675b5bb36aafe0fbd911b8653346a1d4d3d94a09294f17a02eddef180b07a2f6"}
Jul 16 21:02:33.349107 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.349041    2580 scope.go:117] "RemoveContainer" containerID="62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc"
Jul 16 21:02:33.358805 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.358787    2580 scope.go:117] "RemoveContainer" containerID="62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc"
Jul 16 21:02:33.359099 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:02:33.359077    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc\": container with ID starting with 62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc not found: ID does not exist" containerID="62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc"
Jul 16 21:02:33.359175 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.359111    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc"} err="failed to get container status \"62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc\": rpc error: code = NotFound desc = could not find container \"62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc\": container with ID starting with 62b29d0ce685c4b934aa03f7fdb7bc363b5ad551d11bb082609a91470adc6abc not found: ID does not exist"
Jul 16 21:02:33.371161 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.371119    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"]
Jul 16 21:02:33.373567 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.373537    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25df27a3-55bbb8cb86-mcd7b"]
Jul 16 21:02:33.498868 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:33.498832    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e8f28e-ee8c-4769-9898-e19348d873cd" path="/var/lib/kubelet/pods/41e8f28e-ee8c-4769-9898-e19348d873cd/volumes"
Jul 16 21:02:46.614102 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.614070    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"]
Jul 16 21:02:46.614461 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.614448    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e8f28e-ee8c-4769-9898-e19348d873cd" containerName="maas-api"
Jul 16 21:02:46.614497 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.614461    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e8f28e-ee8c-4769-9898-e19348d873cd" containerName="maas-api"
Jul 16 21:02:46.614602 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.614589    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e8f28e-ee8c-4769-9898-e19348d873cd" containerName="maas-api"
Jul 16 21:02:46.617696 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.617677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:46.620205 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.620186    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f05f95fc-serving-cert\""
Jul 16 21:02:46.628637 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.628593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"]
Jul 16 21:02:46.725182 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.725139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:46.725351 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.725191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx66x\" (UniqueName: \"kubernetes.io/projected/eaefa443-1d23-4e82-8578-63637b5c74c6-kube-api-access-dx66x\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:46.725351 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.725225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaefa443-1d23-4e82-8578-63637b5c74c6-openshift-service-ca\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:46.826496 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.826453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:46.826688 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.826532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx66x\" (UniqueName: \"kubernetes.io/projected/eaefa443-1d23-4e82-8578-63637b5c74c6-kube-api-access-dx66x\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:46.826688 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.826570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaefa443-1d23-4e82-8578-63637b5c74c6-openshift-service-ca\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:46.826688 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:02:46.826608    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-serving-cert: secret "maas-api-e2e-derive-f05f95fc-serving-cert" not found
Jul 16 21:02:46.826688 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:02:46.826681    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls podName:eaefa443-1d23-4e82-8578-63637b5c74c6 nodeName:}" failed. No retries permitted until 2026-07-16 21:02:47.326662464 +0000 UTC m=+1774.429941426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls") pod "maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6" (UID: "eaefa443-1d23-4e82-8578-63637b5c74c6") : secret "maas-api-e2e-derive-f05f95fc-serving-cert" not found
Jul 16 21:02:46.827235 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.827218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaefa443-1d23-4e82-8578-63637b5c74c6-openshift-service-ca\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:46.836046 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:46.836024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx66x\" (UniqueName: \"kubernetes.io/projected/eaefa443-1d23-4e82-8578-63637b5c74c6-kube-api-access-dx66x\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:47.331928 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:47.331886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:47.334689 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:47.334656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls\") pod \"maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:47.529770 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:47.529730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:47.679555 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:47.679347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"]
Jul 16 21:02:47.682324 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:02:47.682294    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaefa443_1d23_4e82_8578_63637b5c74c6.slice/crio-f15c9a8448f274e87842171ed21c7a889c28feadeb77284c83628e7f0a3496ca WatchSource:0}: Error finding container f15c9a8448f274e87842171ed21c7a889c28feadeb77284c83628e7f0a3496ca: Status 404 returned error can't find the container with id f15c9a8448f274e87842171ed21c7a889c28feadeb77284c83628e7f0a3496ca
Jul 16 21:02:48.418396 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:48.418360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6" event={"ID":"eaefa443-1d23-4e82-8578-63637b5c74c6","Type":"ContainerStarted","Data":"f15c9a8448f274e87842171ed21c7a889c28feadeb77284c83628e7f0a3496ca"}
Jul 16 21:02:51.433080 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:51.433041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6" event={"ID":"eaefa443-1d23-4e82-8578-63637b5c74c6","Type":"ContainerStarted","Data":"747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05"}
Jul 16 21:02:51.433548 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:51.433228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:02:51.449600 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:51.449555    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6" podStartSLOduration=2.099779769 podStartE2EDuration="5.449539724s" totalImagesPullingTime="3.349759955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:02:51.44863708 +0000 UTC m=+1778.551916069" watchObservedRunningTime="2026-07-16 21:02:51.449539724 +0000 UTC m=+1778.552818703"
Jul 16 21:02:57.442631 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:02:57.442600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:03:13.450283 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:03:13.450251    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 21:03:13.451752 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:03:13.451733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 21:04:00.993384 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:00.993348    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"]
Jul 16 21:04:00.993815 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:00.993613    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6" podUID="eaefa443-1d23-4e82-8578-63637b5c74c6" containerName="maas-api" containerID="cri-o://747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05" gracePeriod=30
Jul 16 21:04:01.245953 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.245885    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:04:01.315140 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.315107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx66x\" (UniqueName: \"kubernetes.io/projected/eaefa443-1d23-4e82-8578-63637b5c74c6-kube-api-access-dx66x\") pod \"eaefa443-1d23-4e82-8578-63637b5c74c6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") "
Jul 16 21:04:01.315305 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.315189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls\") pod \"eaefa443-1d23-4e82-8578-63637b5c74c6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") "
Jul 16 21:04:01.315305 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.315247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaefa443-1d23-4e82-8578-63637b5c74c6-openshift-service-ca\") pod \"eaefa443-1d23-4e82-8578-63637b5c74c6\" (UID: \"eaefa443-1d23-4e82-8578-63637b5c74c6\") "
Jul 16 21:04:01.315679 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.315654    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaefa443-1d23-4e82-8578-63637b5c74c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eaefa443-1d23-4e82-8578-63637b5c74c6" (UID: "eaefa443-1d23-4e82-8578-63637b5c74c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:04:01.317076 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.317042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaefa443-1d23-4e82-8578-63637b5c74c6-kube-api-access-dx66x" (OuterVolumeSpecName: "kube-api-access-dx66x") pod "eaefa443-1d23-4e82-8578-63637b5c74c6" (UID: "eaefa443-1d23-4e82-8578-63637b5c74c6"). InnerVolumeSpecName "kube-api-access-dx66x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:04:01.317235 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.317214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eaefa443-1d23-4e82-8578-63637b5c74c6" (UID: "eaefa443-1d23-4e82-8578-63637b5c74c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:04:01.416142 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.416094    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaefa443-1d23-4e82-8578-63637b5c74c6-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:04:01.416142 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.416138    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx66x\" (UniqueName: \"kubernetes.io/projected/eaefa443-1d23-4e82-8578-63637b5c74c6-kube-api-access-dx66x\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:04:01.416142 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.416149    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaefa443-1d23-4e82-8578-63637b5c74c6-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:04:01.758874 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.758778    2580 generic.go:356] "Generic (PLEG): container finished" podID="eaefa443-1d23-4e82-8578-63637b5c74c6" containerID="747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05" exitCode=0
Jul 16 21:04:01.758874 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.758831    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"
Jul 16 21:04:01.759063 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.758863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6" event={"ID":"eaefa443-1d23-4e82-8578-63637b5c74c6","Type":"ContainerDied","Data":"747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05"}
Jul 16 21:04:01.759063 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.758920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6" event={"ID":"eaefa443-1d23-4e82-8578-63637b5c74c6","Type":"ContainerDied","Data":"f15c9a8448f274e87842171ed21c7a889c28feadeb77284c83628e7f0a3496ca"}
Jul 16 21:04:01.759063 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.758941    2580 scope.go:117] "RemoveContainer" containerID="747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05"
Jul 16 21:04:01.767253 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.767218    2580 scope.go:117] "RemoveContainer" containerID="747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05"
Jul 16 21:04:01.767555 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:04:01.767530    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05\": container with ID starting with 747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05 not found: ID does not exist" containerID="747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05"
Jul 16 21:04:01.767616 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.767563    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05"} err="failed to get container status \"747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05\": rpc error: code = NotFound desc = could not find container \"747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05\": container with ID starting with 747f2901e5d0e7a4bea171fe8f54d9e5e528f28dec5fec04723e05258c70cc05 not found: ID does not exist"
Jul 16 21:04:01.777680 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.777653    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"]
Jul 16 21:04:01.784089 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:01.784062    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f05f95fc-86ddd85c55-v6ff6"]
Jul 16 21:04:03.498696 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:04:03.498662    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaefa443-1d23-4e82-8578-63637b5c74c6" path="/var/lib/kubelet/pods/eaefa443-1d23-4e82-8578-63637b5c74c6/volumes"
Jul 16 21:05:58.063132 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.063092    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"]
Jul 16 21:05:58.063679 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.063448    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaefa443-1d23-4e82-8578-63637b5c74c6" containerName="maas-api"
Jul 16 21:05:58.063679 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.063459    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaefa443-1d23-4e82-8578-63637b5c74c6" containerName="maas-api"
Jul 16 21:05:58.063679 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.063525    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaefa443-1d23-4e82-8578-63637b5c74c6" containerName="maas-api"
Jul 16 21:05:58.066611 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.066591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.069530 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.069494    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a3e438d9-serving-cert\""
Jul 16 21:05:58.082010 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.081985    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"]
Jul 16 21:05:58.225588 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.225546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0971ece2-8d00-4643-a934-968a7af844f8-maas-api-tls\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.225769 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.225612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5qc\" (UniqueName: \"kubernetes.io/projected/0971ece2-8d00-4643-a934-968a7af844f8-kube-api-access-mg5qc\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.225769 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.225672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0971ece2-8d00-4643-a934-968a7af844f8-openshift-service-ca\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.327087 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.327002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0971ece2-8d00-4643-a934-968a7af844f8-maas-api-tls\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.327087 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.327051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5qc\" (UniqueName: \"kubernetes.io/projected/0971ece2-8d00-4643-a934-968a7af844f8-kube-api-access-mg5qc\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.327087 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.327086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0971ece2-8d00-4643-a934-968a7af844f8-openshift-service-ca\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.327817 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.327798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0971ece2-8d00-4643-a934-968a7af844f8-openshift-service-ca\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.329460 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.329442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0971ece2-8d00-4643-a934-968a7af844f8-maas-api-tls\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.339998 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.339975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5qc\" (UniqueName: \"kubernetes.io/projected/0971ece2-8d00-4643-a934-968a7af844f8-kube-api-access-mg5qc\") pod \"maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.377125 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.377094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:05:58.522438 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.522406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"]
Jul 16 21:05:58.524339 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:05:58.524310    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0971ece2_8d00_4643_a934_968a7af844f8.slice/crio-f6fec7a7b2f888d5eafa37316ea31e4c4ec39dcc3a86837662990a2d8684966d WatchSource:0}: Error finding container f6fec7a7b2f888d5eafa37316ea31e4c4ec39dcc3a86837662990a2d8684966d: Status 404 returned error can't find the container with id f6fec7a7b2f888d5eafa37316ea31e4c4ec39dcc3a86837662990a2d8684966d
Jul 16 21:05:58.526216 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:58.526197    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:05:59.287373 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:05:59.287318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg" event={"ID":"0971ece2-8d00-4643-a934-968a7af844f8","Type":"ContainerStarted","Data":"f6fec7a7b2f888d5eafa37316ea31e4c4ec39dcc3a86837662990a2d8684966d"}
Jul 16 21:06:00.293797 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:00.293759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg" event={"ID":"0971ece2-8d00-4643-a934-968a7af844f8","Type":"ContainerStarted","Data":"dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703"}
Jul 16 21:06:00.294170 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:00.293840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:06:00.313053 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:00.313001    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg" podStartSLOduration=1.4197965940000001 podStartE2EDuration="2.312975351s" totalImagesPullingTime="893.178757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:06:00.31201402 +0000 UTC m=+1967.415293005" watchObservedRunningTime="2026-07-16 21:06:00.312975351 +0000 UTC m=+1967.416254337"
Jul 16 21:06:06.304807 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:06.304771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:06:19.687613 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:19.687567    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"]
Jul 16 21:06:19.688035 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:19.687969    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg" podUID="0971ece2-8d00-4643-a934-968a7af844f8" containerName="maas-api" containerID="cri-o://dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703" gracePeriod=30
Jul 16 21:06:19.933631 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:19.933605    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:06:20.012966 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.012873    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0971ece2-8d00-4643-a934-968a7af844f8-openshift-service-ca\") pod \"0971ece2-8d00-4643-a934-968a7af844f8\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") "
Jul 16 21:06:20.012966 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.012958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0971ece2-8d00-4643-a934-968a7af844f8-maas-api-tls\") pod \"0971ece2-8d00-4643-a934-968a7af844f8\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") "
Jul 16 21:06:20.013189 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.013022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg5qc\" (UniqueName: \"kubernetes.io/projected/0971ece2-8d00-4643-a934-968a7af844f8-kube-api-access-mg5qc\") pod \"0971ece2-8d00-4643-a934-968a7af844f8\" (UID: \"0971ece2-8d00-4643-a934-968a7af844f8\") "
Jul 16 21:06:20.013236 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.013192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0971ece2-8d00-4643-a934-968a7af844f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0971ece2-8d00-4643-a934-968a7af844f8" (UID: "0971ece2-8d00-4643-a934-968a7af844f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:06:20.013295 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.013265    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0971ece2-8d00-4643-a934-968a7af844f8-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:06:20.015084 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.015058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0971ece2-8d00-4643-a934-968a7af844f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0971ece2-8d00-4643-a934-968a7af844f8" (UID: "0971ece2-8d00-4643-a934-968a7af844f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:06:20.015179 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.015099    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0971ece2-8d00-4643-a934-968a7af844f8-kube-api-access-mg5qc" (OuterVolumeSpecName: "kube-api-access-mg5qc") pod "0971ece2-8d00-4643-a934-968a7af844f8" (UID: "0971ece2-8d00-4643-a934-968a7af844f8"). InnerVolumeSpecName "kube-api-access-mg5qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:06:20.114221 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.114177    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0971ece2-8d00-4643-a934-968a7af844f8-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:06:20.114221 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.114214    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5qc\" (UniqueName: \"kubernetes.io/projected/0971ece2-8d00-4643-a934-968a7af844f8-kube-api-access-mg5qc\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:06:20.387183 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.387153    2580 generic.go:356] "Generic (PLEG): container finished" podID="0971ece2-8d00-4643-a934-968a7af844f8" containerID="dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703" exitCode=0
Jul 16 21:06:20.387344 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.387191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg" event={"ID":"0971ece2-8d00-4643-a934-968a7af844f8","Type":"ContainerDied","Data":"dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703"}
Jul 16 21:06:20.387344 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.387200    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"
Jul 16 21:06:20.387344 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.387233    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg" event={"ID":"0971ece2-8d00-4643-a934-968a7af844f8","Type":"ContainerDied","Data":"f6fec7a7b2f888d5eafa37316ea31e4c4ec39dcc3a86837662990a2d8684966d"}
Jul 16 21:06:20.387344 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.387253    2580 scope.go:117] "RemoveContainer" containerID="dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703"
Jul 16 21:06:20.396068 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.396051    2580 scope.go:117] "RemoveContainer" containerID="dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703"
Jul 16 21:06:20.396336 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:06:20.396315    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703\": container with ID starting with dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703 not found: ID does not exist" containerID="dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703"
Jul 16 21:06:20.396398 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.396343    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703"} err="failed to get container status \"dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703\": rpc error: code = NotFound desc = could not find container \"dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703\": container with ID starting with dc63ad7137311854e84cd948648d14ce3132e33aae4aca1cf632527ee939b703 not found: ID does not exist"
Jul 16 21:06:20.411734 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.411700    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"]
Jul 16 21:06:20.418783 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:20.418757    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3e438d9-d8f9bf644-dvfkg"]
Jul 16 21:06:21.498795 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:21.498761    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0971ece2-8d00-4643-a934-968a7af844f8" path="/var/lib/kubelet/pods/0971ece2-8d00-4643-a934-968a7af844f8/volumes"
Jul 16 21:06:54.212757 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.212724    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"]
Jul 16 21:06:54.213215 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.213060    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0971ece2-8d00-4643-a934-968a7af844f8" containerName="maas-api"
Jul 16 21:06:54.213215 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.213071    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0971ece2-8d00-4643-a934-968a7af844f8" containerName="maas-api"
Jul 16 21:06:54.213215 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.213142    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0971ece2-8d00-4643-a934-968a7af844f8" containerName="maas-api"
Jul 16 21:06:54.216067 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.216046    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.218747 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.218728    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-27a1e9ed-serving-cert\""
Jul 16 21:06:54.229419 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.229392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"]
Jul 16 21:06:54.304205 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.304172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.304384 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.304212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtlc5\" (UniqueName: \"kubernetes.io/projected/8f4095c4-5d33-45ad-b42d-4f62e6d31739-kube-api-access-dtlc5\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.304384 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.304286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4095c4-5d33-45ad-b42d-4f62e6d31739-openshift-service-ca\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.405167 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.405133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4095c4-5d33-45ad-b42d-4f62e6d31739-openshift-service-ca\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.405360 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.405204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.405360 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.405237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtlc5\" (UniqueName: \"kubernetes.io/projected/8f4095c4-5d33-45ad-b42d-4f62e6d31739-kube-api-access-dtlc5\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.405495 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:06:54.405383    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-serving-cert: secret "maas-api-e2e-mt-27a1e9ed-serving-cert" not found
Jul 16 21:06:54.405495 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:06:54.405482    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls podName:8f4095c4-5d33-45ad-b42d-4f62e6d31739 nodeName:}" failed. No retries permitted until 2026-07-16 21:06:54.905458092 +0000 UTC m=+2022.008737058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls") pod "maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv" (UID: "8f4095c4-5d33-45ad-b42d-4f62e6d31739") : secret "maas-api-e2e-mt-27a1e9ed-serving-cert" not found
Jul 16 21:06:54.405860 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.405830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4095c4-5d33-45ad-b42d-4f62e6d31739-openshift-service-ca\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.416834 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.416802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtlc5\" (UniqueName: \"kubernetes.io/projected/8f4095c4-5d33-45ad-b42d-4f62e6d31739-kube-api-access-dtlc5\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.911150 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.911115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:54.913570 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:54.913548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls\") pod \"maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:55.125933 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:55.125891    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:55.264957 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:55.264745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"]
Jul 16 21:06:55.267076 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:06:55.267043    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4095c4_5d33_45ad_b42d_4f62e6d31739.slice/crio-c7fc9ae7e3509e6ffcd64628c65af66652e9561e2d5c4f28ff643a79265c104b WatchSource:0}: Error finding container c7fc9ae7e3509e6ffcd64628c65af66652e9561e2d5c4f28ff643a79265c104b: Status 404 returned error can't find the container with id c7fc9ae7e3509e6ffcd64628c65af66652e9561e2d5c4f28ff643a79265c104b
Jul 16 21:06:55.540972 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:55.540883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv" event={"ID":"8f4095c4-5d33-45ad-b42d-4f62e6d31739","Type":"ContainerStarted","Data":"c7fc9ae7e3509e6ffcd64628c65af66652e9561e2d5c4f28ff643a79265c104b"}
Jul 16 21:06:57.550230 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:57.550195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv" event={"ID":"8f4095c4-5d33-45ad-b42d-4f62e6d31739","Type":"ContainerStarted","Data":"5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe"}
Jul 16 21:06:57.550657 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:57.550309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:06:57.569690 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:06:57.569634    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv" podStartSLOduration=1.9637398560000001 podStartE2EDuration="3.569615366s" totalImagesPullingTime="1.60587551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:06:57.568494021 +0000 UTC m=+2024.671773009" watchObservedRunningTime="2026-07-16 21:06:57.569615366 +0000 UTC m=+2024.672894350"
Jul 16 21:07:03.561464 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:03.561429    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:07:05.394808 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.394773    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"]
Jul 16 21:07:05.398708 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.398688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.402693 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.402670    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-27a1e9ed\"/\"e2e-lifecycle-model-27a1e9ed-kserve-self-signed-certs\""
Jul 16 21:07:05.404214 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.404190    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-27a1e9ed\"/\"default-dockercfg-wxf9w\""
Jul 16 21:07:05.404214 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.404206    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-27a1e9ed\"/\"kube-root-ca.crt\""
Jul 16 21:07:05.404409 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.404209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-27a1e9ed\"/\"openshift-service-ca.crt\""
Jul 16 21:07:05.410727 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.410701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"]
Jul 16 21:07:05.510808 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.510760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-model-cache\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.510808 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.510812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tls-certs\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.511014 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.510879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-home\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.511014 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.510918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-dshm\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.511014 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.510992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tmp-dir\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.511114 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.511046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcfzx\" (UniqueName: \"kubernetes.io/projected/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-kube-api-access-wcfzx\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612071 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tmp-dir\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612262 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcfzx\" (UniqueName: \"kubernetes.io/projected/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-kube-api-access-wcfzx\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612262 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-model-cache\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612262 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tls-certs\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612262 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-home\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612262 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-dshm\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612587 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tmp-dir\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612767 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-model-cache\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.612926 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.612909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-home\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.614496 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.614470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-dshm\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.614865 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.614840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tls-certs\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.622842 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.622811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcfzx\" (UniqueName: \"kubernetes.io/projected/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-kube-api-access-wcfzx\") pod \"e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") " pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.707998 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.707910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:05.842318 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:05.842291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"]
Jul 16 21:07:05.844267 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:07:05.844230    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ed5745d_6eb3_4e4b_b58b_d457eaaf790c.slice/crio-4b0d1516769c83bb9c2be0ab09e8b4f51ebf9100d4408d714f875749283174de WatchSource:0}: Error finding container 4b0d1516769c83bb9c2be0ab09e8b4f51ebf9100d4408d714f875749283174de: Status 404 returned error can't find the container with id 4b0d1516769c83bb9c2be0ab09e8b4f51ebf9100d4408d714f875749283174de
Jul 16 21:07:06.596932 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:06.596894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5" event={"ID":"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c","Type":"ContainerStarted","Data":"841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336"}
Jul 16 21:07:06.596932 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:06.596936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5" event={"ID":"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c","Type":"ContainerStarted","Data":"4b0d1516769c83bb9c2be0ab09e8b4f51ebf9100d4408d714f875749283174de"}
Jul 16 21:07:06.615896 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:06.615839    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5" podStartSLOduration=1.615823671 podStartE2EDuration="1.615823671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:07:06.614591006 +0000 UTC m=+2033.717870010" watchObservedRunningTime="2026-07-16 21:07:06.615823671 +0000 UTC m=+2033.719102655"
Jul 16 21:07:15.708398 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:15.708356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:15.708833 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:15.708445    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:15.721145 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:15.721097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:16.653182 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:16.653153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:07:42.025489 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.025389    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"]
Jul 16 21:07:42.026071 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.025734    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv" podUID="8f4095c4-5d33-45ad-b42d-4f62e6d31739" containerName="maas-api" containerID="cri-o://5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe" gracePeriod=30
Jul 16 21:07:42.276708 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.276637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:07:42.330240 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.330197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtlc5\" (UniqueName: \"kubernetes.io/projected/8f4095c4-5d33-45ad-b42d-4f62e6d31739-kube-api-access-dtlc5\") pod \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") "
Jul 16 21:07:42.330400 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.330292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls\") pod \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") "
Jul 16 21:07:42.330400 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.330359    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4095c4-5d33-45ad-b42d-4f62e6d31739-openshift-service-ca\") pod \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\" (UID: \"8f4095c4-5d33-45ad-b42d-4f62e6d31739\") "
Jul 16 21:07:42.330751 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.330729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4095c4-5d33-45ad-b42d-4f62e6d31739-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f4095c4-5d33-45ad-b42d-4f62e6d31739" (UID: "8f4095c4-5d33-45ad-b42d-4f62e6d31739"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:07:42.332465 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.332440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4095c4-5d33-45ad-b42d-4f62e6d31739-kube-api-access-dtlc5" (OuterVolumeSpecName: "kube-api-access-dtlc5") pod "8f4095c4-5d33-45ad-b42d-4f62e6d31739" (UID: "8f4095c4-5d33-45ad-b42d-4f62e6d31739"). InnerVolumeSpecName "kube-api-access-dtlc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:07:42.332610 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.332443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f4095c4-5d33-45ad-b42d-4f62e6d31739" (UID: "8f4095c4-5d33-45ad-b42d-4f62e6d31739"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:07:42.431537 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.431478    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4095c4-5d33-45ad-b42d-4f62e6d31739-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:07:42.431537 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.431524    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4095c4-5d33-45ad-b42d-4f62e6d31739-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:07:42.431537 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.431539    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtlc5\" (UniqueName: \"kubernetes.io/projected/8f4095c4-5d33-45ad-b42d-4f62e6d31739-kube-api-access-dtlc5\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:07:42.765727 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.765693    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f4095c4-5d33-45ad-b42d-4f62e6d31739" containerID="5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe" exitCode=0
Jul 16 21:07:42.765908 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.765746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"
Jul 16 21:07:42.765908 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.765766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv" event={"ID":"8f4095c4-5d33-45ad-b42d-4f62e6d31739","Type":"ContainerDied","Data":"5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe"}
Jul 16 21:07:42.765908 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.765812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv" event={"ID":"8f4095c4-5d33-45ad-b42d-4f62e6d31739","Type":"ContainerDied","Data":"c7fc9ae7e3509e6ffcd64628c65af66652e9561e2d5c4f28ff643a79265c104b"}
Jul 16 21:07:42.765908 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.765834    2580 scope.go:117] "RemoveContainer" containerID="5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe"
Jul 16 21:07:42.776657 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.776635    2580 scope.go:117] "RemoveContainer" containerID="5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe"
Jul 16 21:07:42.777039 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:07:42.777018    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe\": container with ID starting with 5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe not found: ID does not exist" containerID="5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe"
Jul 16 21:07:42.777103 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.777047    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe"} err="failed to get container status \"5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe\": rpc error: code = NotFound desc = could not find container \"5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe\": container with ID starting with 5780b22f60146a1d7ba991d7523cfc28466cc9f91feba893917971460c224bbe not found: ID does not exist"
Jul 16 21:07:42.790364 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.790333    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"]
Jul 16 21:07:42.794639 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:42.794616    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27a1e9ed-5466b98c47-qkcgv"]
Jul 16 21:07:43.504945 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:43.504904    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4095c4-5d33-45ad-b42d-4f62e6d31739" path="/var/lib/kubelet/pods/8f4095c4-5d33-45ad-b42d-4f62e6d31739/volumes"
Jul 16 21:07:54.498855 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:07:54.498817    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"]
Jul 16 21:08:09.519107 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.519069    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5" podUID="1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" containerName="main" containerID="cri-o://841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336" gracePeriod=45
Jul 16 21:08:09.771502 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.771437    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:08:09.887584 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.887554    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tls-certs\") pod \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") "
Jul 16 21:08:09.887743 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.887593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-model-cache\") pod \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") "
Jul 16 21:08:09.887743 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.887640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-dshm\") pod \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") "
Jul 16 21:08:09.887743 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.887718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-home\") pod \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") "
Jul 16 21:08:09.887922 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.887759    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tmp-dir\") pod \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") "
Jul 16 21:08:09.887922 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.887784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcfzx\" (UniqueName: \"kubernetes.io/projected/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-kube-api-access-wcfzx\") pod \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\" (UID: \"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c\") "
Jul 16 21:08:09.888024 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.887934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" (UID: "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:08:09.888024 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.887981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-home" (OuterVolumeSpecName: "home") pod "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" (UID: "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:08:09.888024 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.888000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" (UID: "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:08:09.888178 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.888047    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:08:09.888178 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.888064    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:08:09.888989 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.888958    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" containerID="841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336" exitCode=0
Jul 16 21:08:09.889114 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.889015    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"
Jul 16 21:08:09.889114 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.889041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5" event={"ID":"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c","Type":"ContainerDied","Data":"841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336"}
Jul 16 21:08:09.889114 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.889090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5" event={"ID":"1ed5745d-6eb3-4e4b-b58b-d457eaaf790c","Type":"ContainerDied","Data":"4b0d1516769c83bb9c2be0ab09e8b4f51ebf9100d4408d714f875749283174de"}
Jul 16 21:08:09.889114 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.889114    2580 scope.go:117] "RemoveContainer" containerID="841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336"
Jul 16 21:08:09.890101 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.890062    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" (UID: "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:08:09.890233 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.890206    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-dshm" (OuterVolumeSpecName: "dshm") pod "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" (UID: "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:08:09.890233 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.890218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-kube-api-access-wcfzx" (OuterVolumeSpecName: "kube-api-access-wcfzx") pod "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" (UID: "1ed5745d-6eb3-4e4b-b58b-d457eaaf790c"). InnerVolumeSpecName "kube-api-access-wcfzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:08:09.907026 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.907009    2580 scope.go:117] "RemoveContainer" containerID="841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336"
Jul 16 21:08:09.907338 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:08:09.907315    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336\": container with ID starting with 841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336 not found: ID does not exist" containerID="841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336"
Jul 16 21:08:09.907420 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.907347    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336"} err="failed to get container status \"841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336\": rpc error: code = NotFound desc = could not find container \"841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336\": container with ID starting with 841416cde9e6cc4ca5bd26c544c59be5fc3c849dc19a66ab2bca6fad42889336 not found: ID does not exist"
Jul 16 21:08:09.989047 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.989003    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:08:09.989047 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.989039    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:08:09.989047 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.989049    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:08:09.989268 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:09.989059    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcfzx\" (UniqueName: \"kubernetes.io/projected/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c-kube-api-access-wcfzx\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:08:10.219436 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:10.219401    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"]
Jul 16 21:08:10.226064 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:10.226031    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-27a1e9ed/e2e-lifecycle-model-27a1e9ed-kserve-6df6699695-2qwn5"]
Jul 16 21:08:11.499879 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:11.499843    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" path="/var/lib/kubelet/pods/1ed5745d-6eb3-4e4b-b58b-d457eaaf790c/volumes"
Jul 16 21:08:13.481261 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:13.481230    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 21:08:13.484394 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:08:13.484374    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 21:09:40.198874 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.198841    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"]
Jul 16 21:09:40.199376 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.199222    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" containerName="main"
Jul 16 21:09:40.199376 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.199236    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" containerName="main"
Jul 16 21:09:40.199376 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.199246    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4095c4-5d33-45ad-b42d-4f62e6d31739" containerName="maas-api"
Jul 16 21:09:40.199376 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.199252    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4095c4-5d33-45ad-b42d-4f62e6d31739" containerName="maas-api"
Jul 16 21:09:40.199376 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.199327    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ed5745d-6eb3-4e4b-b58b-d457eaaf790c" containerName="main"
Jul 16 21:09:40.199376 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.199336    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4095c4-5d33-45ad-b42d-4f62e6d31739" containerName="maas-api"
Jul 16 21:09:40.202366 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.202349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.206671 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.206651    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-60ce31-serving-cert\""
Jul 16 21:09:40.214763 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.214735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"]
Jul 16 21:09:40.286927 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.286891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zvs\" (UniqueName: \"kubernetes.io/projected/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-kube-api-access-n5zvs\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.287095 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.286991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.287095 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.287048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.388018 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.387989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zvs\" (UniqueName: \"kubernetes.io/projected/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-kube-api-access-n5zvs\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.388183 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.388076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.388183 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.388100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.388262 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:09:40.388197    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-serving-cert: secret "maas-api-e2e-inf-a-60ce31-serving-cert" not found
Jul 16 21:09:40.388308 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:09:40.388276    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls podName:883367b8-a99f-4c0d-a309-9d5cf9a6f8d4 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:40.888253 +0000 UTC m=+2187.991531963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls") pod "maas-api-e2e-inf-a-60ce31-845664c945-mk29b" (UID: "883367b8-a99f-4c0d-a309-9d5cf9a6f8d4") : secret "maas-api-e2e-inf-a-60ce31-serving-cert" not found
Jul 16 21:09:40.388766 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.388746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.404398 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.404376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zvs\" (UniqueName: \"kubernetes.io/projected/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-kube-api-access-n5zvs\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.891501 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.891469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:40.893956 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:40.893934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-60ce31-845664c945-mk29b\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:41.111998 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:41.111955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:41.319594 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:41.319443    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"]
Jul 16 21:09:41.322320 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:09:41.322291    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883367b8_a99f_4c0d_a309_9d5cf9a6f8d4.slice/crio-aa91e1240dc95035dfa74f197b7f6cce8cef1dee956afd74443c503b822601a0 WatchSource:0}: Error finding container aa91e1240dc95035dfa74f197b7f6cce8cef1dee956afd74443c503b822601a0: Status 404 returned error can't find the container with id aa91e1240dc95035dfa74f197b7f6cce8cef1dee956afd74443c503b822601a0
Jul 16 21:09:42.296183 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:42.296141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b" event={"ID":"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4","Type":"ContainerStarted","Data":"d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7"}
Jul 16 21:09:42.296348 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:42.296206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b" event={"ID":"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4","Type":"ContainerStarted","Data":"aa91e1240dc95035dfa74f197b7f6cce8cef1dee956afd74443c503b822601a0"}
Jul 16 21:09:42.296348 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:42.296286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:42.317449 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:42.317397    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b" podStartSLOduration=1.8489732079999999 podStartE2EDuration="2.317379879s" totalImagesPullingTime="468.406671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:09:42.315850783 +0000 UTC m=+2189.419129769" watchObservedRunningTime="2026-07-16 21:09:42.317379879 +0000 UTC m=+2189.420658854"
Jul 16 21:09:48.306708 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:48.306680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:09:51.280823 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.280792    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"]
Jul 16 21:09:51.292262 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.292235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.295404 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.295374    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-054dd3-serving-cert\""
Jul 16 21:09:51.300487 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.300460    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"]
Jul 16 21:09:51.386930 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.386897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70721e3d-423a-462b-a266-a9f5aaec3e87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.387116 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.386947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.387116 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.386965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrjn\" (UniqueName: \"kubernetes.io/projected/70721e3d-423a-462b-a266-a9f5aaec3e87-kube-api-access-wtrjn\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.488204 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.488168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70721e3d-423a-462b-a266-a9f5aaec3e87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.488378 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.488220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.488378 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.488240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrjn\" (UniqueName: \"kubernetes.io/projected/70721e3d-423a-462b-a266-a9f5aaec3e87-kube-api-access-wtrjn\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.488378 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:09:51.488340    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-serving-cert: secret "maas-api-e2e-inf-b-054dd3-serving-cert" not found
Jul 16 21:09:51.488507 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:09:51.488406    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls podName:70721e3d-423a-462b-a266-a9f5aaec3e87 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:51.988390607 +0000 UTC m=+2199.091669570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls") pod "maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8" (UID: "70721e3d-423a-462b-a266-a9f5aaec3e87") : secret "maas-api-e2e-inf-b-054dd3-serving-cert" not found
Jul 16 21:09:51.488913 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.488895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70721e3d-423a-462b-a266-a9f5aaec3e87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.504213 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.504180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrjn\" (UniqueName: \"kubernetes.io/projected/70721e3d-423a-462b-a266-a9f5aaec3e87-kube-api-access-wtrjn\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.992974 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:51.992914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:51.993159 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:09:51.993082    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-serving-cert: secret "maas-api-e2e-inf-b-054dd3-serving-cert" not found
Jul 16 21:09:51.993203 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:09:51.993160    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls podName:70721e3d-423a-462b-a266-a9f5aaec3e87 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:52.993143052 +0000 UTC m=+2200.096422015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls") pod "maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8" (UID: "70721e3d-423a-462b-a266-a9f5aaec3e87") : secret "maas-api-e2e-inf-b-054dd3-serving-cert" not found
Jul 16 21:09:53.001125 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:53.001078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:53.003525 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:53.003497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls\") pod \"maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:53.103887 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:53.103854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:53.262901 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:53.262870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"]
Jul 16 21:09:53.264446 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:09:53.264421    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70721e3d_423a_462b_a266_a9f5aaec3e87.slice/crio-4ebb3cf985861dc93c8ede6012c6ee4bf4dfe3230846231869865b95fa75b7c8 WatchSource:0}: Error finding container 4ebb3cf985861dc93c8ede6012c6ee4bf4dfe3230846231869865b95fa75b7c8: Status 404 returned error can't find the container with id 4ebb3cf985861dc93c8ede6012c6ee4bf4dfe3230846231869865b95fa75b7c8
Jul 16 21:09:53.345943 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:53.345907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8" event={"ID":"70721e3d-423a-462b-a266-a9f5aaec3e87","Type":"ContainerStarted","Data":"4ebb3cf985861dc93c8ede6012c6ee4bf4dfe3230846231869865b95fa75b7c8"}
Jul 16 21:09:55.356251 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:55.356214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8" event={"ID":"70721e3d-423a-462b-a266-a9f5aaec3e87","Type":"ContainerStarted","Data":"0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544"}
Jul 16 21:09:55.356657 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:55.356303    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:09:55.377089 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:55.377022    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8" podStartSLOduration=3.321288172 podStartE2EDuration="4.376999529s" totalImagesPullingTime="1.055711357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:09:55.375152333 +0000 UTC m=+2202.478431320" watchObservedRunningTime="2026-07-16 21:09:55.376999529 +0000 UTC m=+2202.480278515"
Jul 16 21:09:59.684448 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.684410    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"]
Jul 16 21:09:59.687859 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.687843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.691760 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.691739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-60ce31\"/\"default-dockercfg-2rr2d\""
Jul 16 21:09:59.691864 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.691739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-60ce31\"/\"test-model-60ce31-kserve-self-signed-certs\""
Jul 16 21:09:59.691864 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.691746    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-60ce31\"/\"openshift-service-ca.crt\""
Jul 16 21:09:59.691864 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.691767    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-60ce31\"/\"kube-root-ca.crt\""
Jul 16 21:09:59.696559 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.696537    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"]
Jul 16 21:09:59.764638 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.764603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-model-cache\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.764638 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.764636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-tmp-dir\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.764852 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.764656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff48c224-9360-4009-bf50-5c05001df599-tls-certs\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.764852 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.764735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-home\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.764852 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.764757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-dshm\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.764852 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.764826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxccc\" (UniqueName: \"kubernetes.io/projected/ff48c224-9360-4009-bf50-5c05001df599-kube-api-access-bxccc\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.865789 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.865748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxccc\" (UniqueName: \"kubernetes.io/projected/ff48c224-9360-4009-bf50-5c05001df599-kube-api-access-bxccc\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.865976 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.865815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-model-cache\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.865976 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.865832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-tmp-dir\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.865976 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.865849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff48c224-9360-4009-bf50-5c05001df599-tls-certs\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.865976 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.865892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-home\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.865976 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.865907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-dshm\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.866297 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.866259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-home\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.866297 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.866282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-model-cache\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.866459 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.866313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-tmp-dir\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.868128 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.868106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-dshm\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.868478 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.868460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff48c224-9360-4009-bf50-5c05001df599-tls-certs\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.875699 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.875680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxccc\" (UniqueName: \"kubernetes.io/projected/ff48c224-9360-4009-bf50-5c05001df599-kube-api-access-bxccc\") pod \"test-model-60ce31-kserve-5cbf899b97-dg44r\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") " pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:09:59.997912 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:09:59.997823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:10:00.131652 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:00.131624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"]
Jul 16 21:10:00.134329 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:10:00.134293    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff48c224_9360_4009_bf50_5c05001df599.slice/crio-3328fcc6eb919d78aed9d7925198e7838ca5618687557caa9619c880f66b5a5b WatchSource:0}: Error finding container 3328fcc6eb919d78aed9d7925198e7838ca5618687557caa9619c880f66b5a5b: Status 404 returned error can't find the container with id 3328fcc6eb919d78aed9d7925198e7838ca5618687557caa9619c880f66b5a5b
Jul 16 21:10:00.382958 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:00.382866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r" event={"ID":"ff48c224-9360-4009-bf50-5c05001df599","Type":"ContainerStarted","Data":"b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103"}
Jul 16 21:10:00.382958 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:00.382908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r" event={"ID":"ff48c224-9360-4009-bf50-5c05001df599","Type":"ContainerStarted","Data":"3328fcc6eb919d78aed9d7925198e7838ca5618687557caa9619c880f66b5a5b"}
Jul 16 21:10:00.400450 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:00.400392    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r" podStartSLOduration=1.400376211 podStartE2EDuration="1.400376211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:00.398861257 +0000 UTC m=+2207.502140254" watchObservedRunningTime="2026-07-16 21:10:00.400376211 +0000 UTC m=+2207.503655196"
Jul 16 21:10:01.366186 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:01.366154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:10:09.998405 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:09.998365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:10:09.998820 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:09.998428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:10:10.011037 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:10.011013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:10:10.438638 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:10.438602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:10:43.961460 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:43.961379    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"]
Jul 16 21:10:55.044981 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.044941    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"]
Jul 16 21:10:55.045539 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.045194    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b" podUID="883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" containerName="maas-api" containerID="cri-o://d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7" gracePeriod=30
Jul 16 21:10:55.288760 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.288734    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:10:55.462852 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.462815    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-openshift-service-ca\") pod \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") "
Jul 16 21:10:55.463058 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.462884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls\") pod \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") "
Jul 16 21:10:55.463058 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.462923    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5zvs\" (UniqueName: \"kubernetes.io/projected/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-kube-api-access-n5zvs\") pod \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\" (UID: \"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4\") "
Jul 16 21:10:55.463302 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.463263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" (UID: "883367b8-a99f-4c0d-a309-9d5cf9a6f8d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:10:55.465055 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.465032    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" (UID: "883367b8-a99f-4c0d-a309-9d5cf9a6f8d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:10:55.465157 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.465060    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-kube-api-access-n5zvs" (OuterVolumeSpecName: "kube-api-access-n5zvs") pod "883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" (UID: "883367b8-a99f-4c0d-a309-9d5cf9a6f8d4"). InnerVolumeSpecName "kube-api-access-n5zvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:10:55.564265 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.564216    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:55.564265 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.564257    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zvs\" (UniqueName: \"kubernetes.io/projected/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-kube-api-access-n5zvs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:55.564469 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.564278    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:55.631591 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.631554    2580 generic.go:356] "Generic (PLEG): container finished" podID="883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" containerID="d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7" exitCode=0
Jul 16 21:10:55.631756 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.631619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b" event={"ID":"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4","Type":"ContainerDied","Data":"d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7"}
Jul 16 21:10:55.631756 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.631657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b" event={"ID":"883367b8-a99f-4c0d-a309-9d5cf9a6f8d4","Type":"ContainerDied","Data":"aa91e1240dc95035dfa74f197b7f6cce8cef1dee956afd74443c503b822601a0"}
Jul 16 21:10:55.631756 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.631678    2580 scope.go:117] "RemoveContainer" containerID="d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7"
Jul 16 21:10:55.631756 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.631621    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"
Jul 16 21:10:55.640291 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.640269    2580 scope.go:117] "RemoveContainer" containerID="d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7"
Jul 16 21:10:55.640567 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:10:55.640547    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7\": container with ID starting with d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7 not found: ID does not exist" containerID="d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7"
Jul 16 21:10:55.640626 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.640576    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7"} err="failed to get container status \"d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7\": rpc error: code = NotFound desc = could not find container \"d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7\": container with ID starting with d8928e4fdfbdb44eaac0793bb063a457ec2a5a01d7cca87d23c901bedd0f19b7 not found: ID does not exist"
Jul 16 21:10:55.648937 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.648913    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"]
Jul 16 21:10:55.655135 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:55.655112    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60ce31-845664c945-mk29b"]
Jul 16 21:10:57.498680 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:57.498646    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" path="/var/lib/kubelet/pods/883367b8-a99f-4c0d-a309-9d5cf9a6f8d4/volumes"
Jul 16 21:10:58.982224 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:58.982166    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r" podUID="ff48c224-9360-4009-bf50-5c05001df599" containerName="main" containerID="cri-o://b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103" gracePeriod=45
Jul 16 21:10:59.237260 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.237201    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:10:59.398843 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.398813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-tmp-dir\") pod \"ff48c224-9360-4009-bf50-5c05001df599\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") "
Jul 16 21:10:59.399023 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.398855    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxccc\" (UniqueName: \"kubernetes.io/projected/ff48c224-9360-4009-bf50-5c05001df599-kube-api-access-bxccc\") pod \"ff48c224-9360-4009-bf50-5c05001df599\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") "
Jul 16 21:10:59.399023 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.398887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff48c224-9360-4009-bf50-5c05001df599-tls-certs\") pod \"ff48c224-9360-4009-bf50-5c05001df599\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") "
Jul 16 21:10:59.399023 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.398906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-home\") pod \"ff48c224-9360-4009-bf50-5c05001df599\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") "
Jul 16 21:10:59.399023 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.398922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-dshm\") pod \"ff48c224-9360-4009-bf50-5c05001df599\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") "
Jul 16 21:10:59.399023 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.398964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-model-cache\") pod \"ff48c224-9360-4009-bf50-5c05001df599\" (UID: \"ff48c224-9360-4009-bf50-5c05001df599\") "
Jul 16 21:10:59.399286 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.399178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff48c224-9360-4009-bf50-5c05001df599" (UID: "ff48c224-9360-4009-bf50-5c05001df599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:10:59.399286 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.399195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-home" (OuterVolumeSpecName: "home") pod "ff48c224-9360-4009-bf50-5c05001df599" (UID: "ff48c224-9360-4009-bf50-5c05001df599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:10:59.399444 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.399404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff48c224-9360-4009-bf50-5c05001df599" (UID: "ff48c224-9360-4009-bf50-5c05001df599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:10:59.401167 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.401144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff48c224-9360-4009-bf50-5c05001df599-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff48c224-9360-4009-bf50-5c05001df599" (UID: "ff48c224-9360-4009-bf50-5c05001df599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:10:59.401385 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.401358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-dshm" (OuterVolumeSpecName: "dshm") pod "ff48c224-9360-4009-bf50-5c05001df599" (UID: "ff48c224-9360-4009-bf50-5c05001df599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:10:59.401467 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.401383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff48c224-9360-4009-bf50-5c05001df599-kube-api-access-bxccc" (OuterVolumeSpecName: "kube-api-access-bxccc") pod "ff48c224-9360-4009-bf50-5c05001df599" (UID: "ff48c224-9360-4009-bf50-5c05001df599"). InnerVolumeSpecName "kube-api-access-bxccc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:10:59.500257 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.500180    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:59.500257 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.500206    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxccc\" (UniqueName: \"kubernetes.io/projected/ff48c224-9360-4009-bf50-5c05001df599-kube-api-access-bxccc\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:59.500257 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.500216    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff48c224-9360-4009-bf50-5c05001df599-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:59.500257 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.500224    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:59.500257 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.500231    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:59.500257 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.500238    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff48c224-9360-4009-bf50-5c05001df599-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:10:59.652818 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.652782    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff48c224-9360-4009-bf50-5c05001df599" containerID="b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103" exitCode=0
Jul 16 21:10:59.652987 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.652830    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"
Jul 16 21:10:59.652987 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.652873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r" event={"ID":"ff48c224-9360-4009-bf50-5c05001df599","Type":"ContainerDied","Data":"b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103"}
Jul 16 21:10:59.652987 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.652920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r" event={"ID":"ff48c224-9360-4009-bf50-5c05001df599","Type":"ContainerDied","Data":"3328fcc6eb919d78aed9d7925198e7838ca5618687557caa9619c880f66b5a5b"}
Jul 16 21:10:59.652987 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.652984    2580 scope.go:117] "RemoveContainer" containerID="b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103"
Jul 16 21:10:59.661113 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.661090    2580 scope.go:117] "RemoveContainer" containerID="b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103"
Jul 16 21:10:59.661375 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:10:59.661357    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103\": container with ID starting with b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103 not found: ID does not exist" containerID="b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103"
Jul 16 21:10:59.661424 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.661383    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103"} err="failed to get container status \"b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103\": rpc error: code = NotFound desc = could not find container \"b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103\": container with ID starting with b3a91ba0a1681823adaec8a56e63c4f25e3301a4f4a5e5e8f711a43bfb04b103 not found: ID does not exist"
Jul 16 21:10:59.670233 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.670209    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"]
Jul 16 21:10:59.676892 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:10:59.676871    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-60ce31/test-model-60ce31-kserve-5cbf899b97-dg44r"]
Jul 16 21:11:01.498849 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:01.498818    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff48c224-9360-4009-bf50-5c05001df599" path="/var/lib/kubelet/pods/ff48c224-9360-4009-bf50-5c05001df599/volumes"
Jul 16 21:11:17.430220 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.430179    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"]
Jul 16 21:11:17.430761 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.430390    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8" podUID="70721e3d-423a-462b-a266-a9f5aaec3e87" containerName="maas-api" containerID="cri-o://0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544" gracePeriod=30
Jul 16 21:11:17.677220 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.677191    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:11:17.734546 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.734438    2580 generic.go:356] "Generic (PLEG): container finished" podID="70721e3d-423a-462b-a266-a9f5aaec3e87" containerID="0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544" exitCode=0
Jul 16 21:11:17.734546 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.734504    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"
Jul 16 21:11:17.734731 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.734568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8" event={"ID":"70721e3d-423a-462b-a266-a9f5aaec3e87","Type":"ContainerDied","Data":"0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544"}
Jul 16 21:11:17.734731 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.734661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8" event={"ID":"70721e3d-423a-462b-a266-a9f5aaec3e87","Type":"ContainerDied","Data":"4ebb3cf985861dc93c8ede6012c6ee4bf4dfe3230846231869865b95fa75b7c8"}
Jul 16 21:11:17.734731 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.734684    2580 scope.go:117] "RemoveContainer" containerID="0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544"
Jul 16 21:11:17.742722 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.742703    2580 scope.go:117] "RemoveContainer" containerID="0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544"
Jul 16 21:11:17.742998 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:11:17.742975    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544\": container with ID starting with 0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544 not found: ID does not exist" containerID="0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544"
Jul 16 21:11:17.743063 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.743010    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544"} err="failed to get container status \"0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544\": rpc error: code = NotFound desc = could not find container \"0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544\": container with ID starting with 0191bf35dcac5ef354a7daff17c9dfb5a562ef01142fd7808ef5a4d5a5c4e544 not found: ID does not exist"
Jul 16 21:11:17.752407 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.752388    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls\") pod \"70721e3d-423a-462b-a266-a9f5aaec3e87\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") "
Jul 16 21:11:17.752506 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.752445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70721e3d-423a-462b-a266-a9f5aaec3e87-openshift-service-ca\") pod \"70721e3d-423a-462b-a266-a9f5aaec3e87\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") "
Jul 16 21:11:17.752577 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.752509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtrjn\" (UniqueName: \"kubernetes.io/projected/70721e3d-423a-462b-a266-a9f5aaec3e87-kube-api-access-wtrjn\") pod \"70721e3d-423a-462b-a266-a9f5aaec3e87\" (UID: \"70721e3d-423a-462b-a266-a9f5aaec3e87\") "
Jul 16 21:11:17.752815 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.752792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70721e3d-423a-462b-a266-a9f5aaec3e87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70721e3d-423a-462b-a266-a9f5aaec3e87" (UID: "70721e3d-423a-462b-a266-a9f5aaec3e87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:11:17.754444 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.754419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70721e3d-423a-462b-a266-a9f5aaec3e87" (UID: "70721e3d-423a-462b-a266-a9f5aaec3e87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:11:17.754522 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.754456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70721e3d-423a-462b-a266-a9f5aaec3e87-kube-api-access-wtrjn" (OuterVolumeSpecName: "kube-api-access-wtrjn") pod "70721e3d-423a-462b-a266-a9f5aaec3e87" (UID: "70721e3d-423a-462b-a266-a9f5aaec3e87"). InnerVolumeSpecName "kube-api-access-wtrjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:11:17.853387 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.853351    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70721e3d-423a-462b-a266-a9f5aaec3e87-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:11:17.853387 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.853382    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrjn\" (UniqueName: \"kubernetes.io/projected/70721e3d-423a-462b-a266-a9f5aaec3e87-kube-api-access-wtrjn\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:11:17.853387 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:17.853392    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70721e3d-423a-462b-a266-a9f5aaec3e87-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:11:18.056804 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:18.056774    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"]
Jul 16 21:11:18.061886 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:18.061861    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-054dd3-845f8bbd94-cgqg8"]
Jul 16 21:11:19.498603 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:19.498568    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70721e3d-423a-462b-a266-a9f5aaec3e87" path="/var/lib/kubelet/pods/70721e3d-423a-462b-a266-a9f5aaec3e87/volumes"
Jul 16 21:11:32.900652 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.900614    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"]
Jul 16 21:11:32.901098 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901034    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70721e3d-423a-462b-a266-a9f5aaec3e87" containerName="maas-api"
Jul 16 21:11:32.901098 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901047    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="70721e3d-423a-462b-a266-a9f5aaec3e87" containerName="maas-api"
Jul 16 21:11:32.901098 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901065    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff48c224-9360-4009-bf50-5c05001df599" containerName="main"
Jul 16 21:11:32.901098 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901070    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff48c224-9360-4009-bf50-5c05001df599" containerName="main"
Jul 16 21:11:32.901098 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901078    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" containerName="maas-api"
Jul 16 21:11:32.901098 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901084    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" containerName="maas-api"
Jul 16 21:11:32.901285 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901140    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff48c224-9360-4009-bf50-5c05001df599" containerName="main"
Jul 16 21:11:32.901285 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901149    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="70721e3d-423a-462b-a266-a9f5aaec3e87" containerName="maas-api"
Jul 16 21:11:32.901285 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.901157    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="883367b8-a99f-4c0d-a309-9d5cf9a6f8d4" containerName="maas-api"
Jul 16 21:11:32.904374 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.904353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:32.907643 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.907621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-99f81a-serving-cert\""
Jul 16 21:11:32.917697 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.917672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"]
Jul 16 21:11:32.980765 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.980729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:32.980902 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.980771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4c830a-127d-401e-b310-df7d7d346c36-openshift-service-ca\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:32.980902 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:32.980798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx74g\" (UniqueName: \"kubernetes.io/projected/5f4c830a-127d-401e-b310-df7d7d346c36-kube-api-access-tx74g\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.081413 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.081373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.081413 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.081414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4c830a-127d-401e-b310-df7d7d346c36-openshift-service-ca\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.081672 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.081431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx74g\" (UniqueName: \"kubernetes.io/projected/5f4c830a-127d-401e-b310-df7d7d346c36-kube-api-access-tx74g\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.081672 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:11:33.081565    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-serving-cert: secret "maas-api-e2e-api-a-99f81a-serving-cert" not found
Jul 16 21:11:33.081672 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:11:33.081647    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls podName:5f4c830a-127d-401e-b310-df7d7d346c36 nodeName:}" failed. No retries permitted until 2026-07-16 21:11:33.581627303 +0000 UTC m=+2300.684906267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls") pod "maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r" (UID: "5f4c830a-127d-401e-b310-df7d7d346c36") : secret "maas-api-e2e-api-a-99f81a-serving-cert" not found
Jul 16 21:11:33.082114 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.082096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4c830a-127d-401e-b310-df7d7d346c36-openshift-service-ca\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.098953 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.098925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx74g\" (UniqueName: \"kubernetes.io/projected/5f4c830a-127d-401e-b310-df7d7d346c36-kube-api-access-tx74g\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.586768 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.586731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.589312 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.589290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls\") pod \"maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.815078 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.815041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:33.962227 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.962140    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"]
Jul 16 21:11:33.964713 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:11:33.964677    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4c830a_127d_401e_b310_df7d7d346c36.slice/crio-243d837be869f8a9869bc1cd79d8311a7851d5e6819385e0cc28a6a8adefdba2 WatchSource:0}: Error finding container 243d837be869f8a9869bc1cd79d8311a7851d5e6819385e0cc28a6a8adefdba2: Status 404 returned error can't find the container with id 243d837be869f8a9869bc1cd79d8311a7851d5e6819385e0cc28a6a8adefdba2
Jul 16 21:11:33.966158 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:33.966138    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:11:34.811728 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:34.811645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r" event={"ID":"5f4c830a-127d-401e-b310-df7d7d346c36","Type":"ContainerStarted","Data":"92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0"}
Jul 16 21:11:34.811728 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:34.811705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r" event={"ID":"5f4c830a-127d-401e-b310-df7d7d346c36","Type":"ContainerStarted","Data":"243d837be869f8a9869bc1cd79d8311a7851d5e6819385e0cc28a6a8adefdba2"}
Jul 16 21:11:34.811945 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:34.811762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:34.830238 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:34.830191    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r" podStartSLOduration=2.405841944 podStartE2EDuration="2.830177622s" totalImagesPullingTime="424.335678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:34.828629511 +0000 UTC m=+2301.931908499" watchObservedRunningTime="2026-07-16 21:11:34.830177622 +0000 UTC m=+2301.933456607"
Jul 16 21:11:40.821452 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:40.821422    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:11:40.887999 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:40.887959    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"]
Jul 16 21:11:40.891726 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:40.891703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:40.894362 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:40.894341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a896c7-serving-cert\""
Jul 16 21:11:40.909920 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:40.909888    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"]
Jul 16 21:11:40.945278 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:40.945241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfshs\" (UniqueName: \"kubernetes.io/projected/c56994ab-452e-4d2b-bdd4-0a452530e8db-kube-api-access-lfshs\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:40.945462 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:40.945343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56994ab-452e-4d2b-bdd4-0a452530e8db-openshift-service-ca\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:40.945462 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:40.945443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.046145 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.046105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfshs\" (UniqueName: \"kubernetes.io/projected/c56994ab-452e-4d2b-bdd4-0a452530e8db-kube-api-access-lfshs\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.046145 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.046148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56994ab-452e-4d2b-bdd4-0a452530e8db-openshift-service-ca\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.046356 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.046173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.046356 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:11:41.046296    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-serving-cert: secret "maas-api-e2e-api-b-a896c7-serving-cert" not found
Jul 16 21:11:41.046430 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:11:41.046360    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls podName:c56994ab-452e-4d2b-bdd4-0a452530e8db nodeName:}" failed. No retries permitted until 2026-07-16 21:11:41.54634024 +0000 UTC m=+2308.649619221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls") pod "maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4" (UID: "c56994ab-452e-4d2b-bdd4-0a452530e8db") : secret "maas-api-e2e-api-b-a896c7-serving-cert" not found
Jul 16 21:11:41.046890 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.046872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56994ab-452e-4d2b-bdd4-0a452530e8db-openshift-service-ca\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.059776 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.059744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfshs\" (UniqueName: \"kubernetes.io/projected/c56994ab-452e-4d2b-bdd4-0a452530e8db-kube-api-access-lfshs\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.550838 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.550797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.553266 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.553239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls\") pod \"maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.802931 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.802837    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:41.940553 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:41.940502    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"]
Jul 16 21:11:41.944920 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:11:41.943806    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56994ab_452e_4d2b_bdd4_0a452530e8db.slice/crio-cee0c43be2f5920a5b36bb03fb431185cbb971ea2531dab46988350bf74bd1b1 WatchSource:0}: Error finding container cee0c43be2f5920a5b36bb03fb431185cbb971ea2531dab46988350bf74bd1b1: Status 404 returned error can't find the container with id cee0c43be2f5920a5b36bb03fb431185cbb971ea2531dab46988350bf74bd1b1
Jul 16 21:11:42.849462 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:42.849423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4" event={"ID":"c56994ab-452e-4d2b-bdd4-0a452530e8db","Type":"ContainerStarted","Data":"00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce"}
Jul 16 21:11:42.849645 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:42.849489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:11:42.849645 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:42.849507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4" event={"ID":"c56994ab-452e-4d2b-bdd4-0a452530e8db","Type":"ContainerStarted","Data":"cee0c43be2f5920a5b36bb03fb431185cbb971ea2531dab46988350bf74bd1b1"}
Jul 16 21:11:42.867284 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:42.867240    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4" podStartSLOduration=2.497137765 podStartE2EDuration="2.867225104s" totalImagesPullingTime="370.087339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:42.866476765 +0000 UTC m=+2309.969755817" watchObservedRunningTime="2026-07-16 21:11:42.867225104 +0000 UTC m=+2309.970504090"
Jul 16 21:11:48.858947 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:11:48.858916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:12:08.493825 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.493792    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"]
Jul 16 21:12:08.494343 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.494079    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r" podUID="5f4c830a-127d-401e-b310-df7d7d346c36" containerName="maas-api" containerID="cri-o://92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0" gracePeriod=30
Jul 16 21:12:08.737718 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.737695    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:12:08.781283 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.781213    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls\") pod \"5f4c830a-127d-401e-b310-df7d7d346c36\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") "
Jul 16 21:12:08.781422 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.781305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx74g\" (UniqueName: \"kubernetes.io/projected/5f4c830a-127d-401e-b310-df7d7d346c36-kube-api-access-tx74g\") pod \"5f4c830a-127d-401e-b310-df7d7d346c36\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") "
Jul 16 21:12:08.781422 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.781341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4c830a-127d-401e-b310-df7d7d346c36-openshift-service-ca\") pod \"5f4c830a-127d-401e-b310-df7d7d346c36\" (UID: \"5f4c830a-127d-401e-b310-df7d7d346c36\") "
Jul 16 21:12:08.781696 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.781673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4c830a-127d-401e-b310-df7d7d346c36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f4c830a-127d-401e-b310-df7d7d346c36" (UID: "5f4c830a-127d-401e-b310-df7d7d346c36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:12:08.783333 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.783313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f4c830a-127d-401e-b310-df7d7d346c36" (UID: "5f4c830a-127d-401e-b310-df7d7d346c36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:12:08.783438 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.783358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4c830a-127d-401e-b310-df7d7d346c36-kube-api-access-tx74g" (OuterVolumeSpecName: "kube-api-access-tx74g") pod "5f4c830a-127d-401e-b310-df7d7d346c36" (UID: "5f4c830a-127d-401e-b310-df7d7d346c36"). InnerVolumeSpecName "kube-api-access-tx74g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:12:08.882752 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.882723    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx74g\" (UniqueName: \"kubernetes.io/projected/5f4c830a-127d-401e-b310-df7d7d346c36-kube-api-access-tx74g\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:12:08.882752 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.882748    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4c830a-127d-401e-b310-df7d7d346c36-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:12:08.882752 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.882757    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4c830a-127d-401e-b310-df7d7d346c36-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:12:08.973824 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.973796    2580 generic.go:356] "Generic (PLEG): container finished" podID="5f4c830a-127d-401e-b310-df7d7d346c36" containerID="92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0" exitCode=0
Jul 16 21:12:08.973979 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.973842    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"
Jul 16 21:12:08.973979 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.973890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r" event={"ID":"5f4c830a-127d-401e-b310-df7d7d346c36","Type":"ContainerDied","Data":"92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0"}
Jul 16 21:12:08.973979 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.973939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r" event={"ID":"5f4c830a-127d-401e-b310-df7d7d346c36","Type":"ContainerDied","Data":"243d837be869f8a9869bc1cd79d8311a7851d5e6819385e0cc28a6a8adefdba2"}
Jul 16 21:12:08.973979 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.973960    2580 scope.go:117] "RemoveContainer" containerID="92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0"
Jul 16 21:12:08.988336 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.988311    2580 scope.go:117] "RemoveContainer" containerID="92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0"
Jul 16 21:12:08.988774 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:12:08.988733    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0\": container with ID starting with 92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0 not found: ID does not exist" containerID="92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0"
Jul 16 21:12:08.988883 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.988782    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0"} err="failed to get container status \"92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0\": rpc error: code = NotFound desc = could not find container \"92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0\": container with ID starting with 92a06bd1ed1d922a2f047bfcd63ec0768df6f351fae9fd147784fbec09fdeca0 not found: ID does not exist"
Jul 16 21:12:08.998382 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:08.998356    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"]
Jul 16 21:12:09.001898 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:09.001878    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-99f81a-67b5c7469b-bkr5r"]
Jul 16 21:12:09.498115 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:09.498081    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f4c830a-127d-401e-b310-df7d7d346c36" path="/var/lib/kubelet/pods/5f4c830a-127d-401e-b310-df7d7d346c36/volumes"
Jul 16 21:12:35.389257 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.389219    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"]
Jul 16 21:12:35.389904 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.389499    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4" podUID="c56994ab-452e-4d2b-bdd4-0a452530e8db" containerName="maas-api" containerID="cri-o://00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce" gracePeriod=30
Jul 16 21:12:35.637972 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.637941    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:12:35.723371 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.723288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls\") pod \"c56994ab-452e-4d2b-bdd4-0a452530e8db\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") "
Jul 16 21:12:35.723371 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.723346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfshs\" (UniqueName: \"kubernetes.io/projected/c56994ab-452e-4d2b-bdd4-0a452530e8db-kube-api-access-lfshs\") pod \"c56994ab-452e-4d2b-bdd4-0a452530e8db\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") "
Jul 16 21:12:35.723629 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.723406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56994ab-452e-4d2b-bdd4-0a452530e8db-openshift-service-ca\") pod \"c56994ab-452e-4d2b-bdd4-0a452530e8db\" (UID: \"c56994ab-452e-4d2b-bdd4-0a452530e8db\") "
Jul 16 21:12:35.723832 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.723799    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56994ab-452e-4d2b-bdd4-0a452530e8db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c56994ab-452e-4d2b-bdd4-0a452530e8db" (UID: "c56994ab-452e-4d2b-bdd4-0a452530e8db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:12:35.725436 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.725412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56994ab-452e-4d2b-bdd4-0a452530e8db-kube-api-access-lfshs" (OuterVolumeSpecName: "kube-api-access-lfshs") pod "c56994ab-452e-4d2b-bdd4-0a452530e8db" (UID: "c56994ab-452e-4d2b-bdd4-0a452530e8db"). InnerVolumeSpecName "kube-api-access-lfshs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:12:35.725538 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.725490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c56994ab-452e-4d2b-bdd4-0a452530e8db" (UID: "c56994ab-452e-4d2b-bdd4-0a452530e8db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:12:35.824559 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.824499    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56994ab-452e-4d2b-bdd4-0a452530e8db-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:12:35.824559 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.824555    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56994ab-452e-4d2b-bdd4-0a452530e8db-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:12:35.824559 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:35.824565    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfshs\" (UniqueName: \"kubernetes.io/projected/c56994ab-452e-4d2b-bdd4-0a452530e8db-kube-api-access-lfshs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:12:36.105926 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.105838    2580 generic.go:356] "Generic (PLEG): container finished" podID="c56994ab-452e-4d2b-bdd4-0a452530e8db" containerID="00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce" exitCode=0
Jul 16 21:12:36.105926 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.105893    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"
Jul 16 21:12:36.106101 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.105897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4" event={"ID":"c56994ab-452e-4d2b-bdd4-0a452530e8db","Type":"ContainerDied","Data":"00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce"}
Jul 16 21:12:36.106101 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.106004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4" event={"ID":"c56994ab-452e-4d2b-bdd4-0a452530e8db","Type":"ContainerDied","Data":"cee0c43be2f5920a5b36bb03fb431185cbb971ea2531dab46988350bf74bd1b1"}
Jul 16 21:12:36.106101 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.106027    2580 scope.go:117] "RemoveContainer" containerID="00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce"
Jul 16 21:12:36.115048 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.115029    2580 scope.go:117] "RemoveContainer" containerID="00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce"
Jul 16 21:12:36.115295 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:12:36.115275    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce\": container with ID starting with 00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce not found: ID does not exist" containerID="00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce"
Jul 16 21:12:36.115332 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.115304    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce"} err="failed to get container status \"00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce\": rpc error: code = NotFound desc = could not find container \"00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce\": container with ID starting with 00e11fcd0285dd1a28dd65b672364a070bad59b5c8b15f27b17ea9354f0187ce not found: ID does not exist"
Jul 16 21:12:36.128813 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.128783    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"]
Jul 16 21:12:36.135597 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:36.135576    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a896c7-5fb6585857-bd8q4"]
Jul 16 21:12:37.498034 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:37.498001    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56994ab-452e-4d2b-bdd4-0a452530e8db" path="/var/lib/kubelet/pods/c56994ab-452e-4d2b-bdd4-0a452530e8db/volumes"
Jul 16 21:12:51.393974 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.393935    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"]
Jul 16 21:12:51.394569 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.394475    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56994ab-452e-4d2b-bdd4-0a452530e8db" containerName="maas-api"
Jul 16 21:12:51.394569 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.394494    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56994ab-452e-4d2b-bdd4-0a452530e8db" containerName="maas-api"
Jul 16 21:12:51.394569 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.394530    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f4c830a-127d-401e-b310-df7d7d346c36" containerName="maas-api"
Jul 16 21:12:51.394569 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.394540    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f4c830a-127d-401e-b310-df7d7d346c36" containerName="maas-api"
Jul 16 21:12:51.394791 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.394639    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f4c830a-127d-401e-b310-df7d7d346c36" containerName="maas-api"
Jul 16 21:12:51.394791 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.394657    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56994ab-452e-4d2b-bdd4-0a452530e8db" containerName="maas-api"
Jul 16 21:12:51.397689 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.397666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:51.400365 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.400340    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-09402f-serving-cert\""
Jul 16 21:12:51.410120 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.410097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"]
Jul 16 21:12:51.465363 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.465325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:51.465573 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.465389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60fc00-d091-462b-b897-f1b1b30d5361-openshift-service-ca\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:51.465573 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.465450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgms\" (UniqueName: \"kubernetes.io/projected/dc60fc00-d091-462b-b897-f1b1b30d5361-kube-api-access-wxgms\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:51.566309 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.566262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60fc00-d091-462b-b897-f1b1b30d5361-openshift-service-ca\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:51.566499 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.566331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgms\" (UniqueName: \"kubernetes.io/projected/dc60fc00-d091-462b-b897-f1b1b30d5361-kube-api-access-wxgms\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:51.566499 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.566390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:51.566648 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:12:51.566629    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-serving-cert: secret "maas-api-e2e-shared-a-09402f-serving-cert" not found
Jul 16 21:12:51.566740 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:12:51.566726    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls podName:dc60fc00-d091-462b-b897-f1b1b30d5361 nodeName:}" failed. No retries permitted until 2026-07-16 21:12:52.066702295 +0000 UTC m=+2379.169981281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls") pod "maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk" (UID: "dc60fc00-d091-462b-b897-f1b1b30d5361") : secret "maas-api-e2e-shared-a-09402f-serving-cert" not found
Jul 16 21:12:51.572533 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.567364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60fc00-d091-462b-b897-f1b1b30d5361-openshift-service-ca\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:51.577710 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:51.577681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgms\" (UniqueName: \"kubernetes.io/projected/dc60fc00-d091-462b-b897-f1b1b30d5361-kube-api-access-wxgms\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:52.072159 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:52.072115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:52.074601 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:52.074581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls\") pod \"maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:52.308106 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:52.308066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:52.442830 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:52.442801    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"]
Jul 16 21:12:52.444580 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:12:52.444551    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc60fc00_d091_462b_b897_f1b1b30d5361.slice/crio-efa4bbda8a5b092689f1b6441ddf4d013170530d6dd98ef82d2d820614972006 WatchSource:0}: Error finding container efa4bbda8a5b092689f1b6441ddf4d013170530d6dd98ef82d2d820614972006: Status 404 returned error can't find the container with id efa4bbda8a5b092689f1b6441ddf4d013170530d6dd98ef82d2d820614972006
Jul 16 21:12:53.182728 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:53.182692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk" event={"ID":"dc60fc00-d091-462b-b897-f1b1b30d5361","Type":"ContainerStarted","Data":"18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877"}
Jul 16 21:12:53.182946 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:53.182744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk" event={"ID":"dc60fc00-d091-462b-b897-f1b1b30d5361","Type":"ContainerStarted","Data":"efa4bbda8a5b092689f1b6441ddf4d013170530d6dd98ef82d2d820614972006"}
Jul 16 21:12:53.182946 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:53.182770    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:53.201033 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:53.200981    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk" podStartSLOduration=1.600266162 podStartE2EDuration="2.200962134s" totalImagesPullingTime="600.695972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:12:53.199280553 +0000 UTC m=+2380.302559540" watchObservedRunningTime="2026-07-16 21:12:53.200962134 +0000 UTC m=+2380.304241138"
Jul 16 21:12:59.192830 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.192798    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:12:59.411117 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.411089    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"]
Jul 16 21:12:59.414533 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.414492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:12:59.417127 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.417107    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-69d8e7-serving-cert\""
Jul 16 21:12:59.426836 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.426807    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"]
Jul 16 21:12:59.436938 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.436914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:12:59.437035 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.436962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxm5\" (UniqueName: \"kubernetes.io/projected/dbd8cdb6-e206-4e82-822d-09a73b958361-kube-api-access-lqxm5\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:12:59.437035 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.436991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd8cdb6-e206-4e82-822d-09a73b958361-openshift-service-ca\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:12:59.538183 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.538110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:12:59.538183 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.538156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxm5\" (UniqueName: \"kubernetes.io/projected/dbd8cdb6-e206-4e82-822d-09a73b958361-kube-api-access-lqxm5\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:12:59.538183 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.538181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd8cdb6-e206-4e82-822d-09a73b958361-openshift-service-ca\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:12:59.538411 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:12:59.538270    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-serving-cert: secret "maas-api-e2e-shared-b-69d8e7-serving-cert" not found
Jul 16 21:12:59.538411 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:12:59.538350    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls podName:dbd8cdb6-e206-4e82-822d-09a73b958361 nodeName:}" failed. No retries permitted until 2026-07-16 21:13:00.038329961 +0000 UTC m=+2387.141608946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls") pod "maas-api-e2e-shared-b-69d8e7-66d99547d9-44428" (UID: "dbd8cdb6-e206-4e82-822d-09a73b958361") : secret "maas-api-e2e-shared-b-69d8e7-serving-cert" not found
Jul 16 21:12:59.538905 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.538888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd8cdb6-e206-4e82-822d-09a73b958361-openshift-service-ca\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:12:59.549188 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:12:59.549167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxm5\" (UniqueName: \"kubernetes.io/projected/dbd8cdb6-e206-4e82-822d-09a73b958361-kube-api-access-lqxm5\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:13:00.042570 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:00.042533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:13:00.044905 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:00.044884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls\") pod \"maas-api-e2e-shared-b-69d8e7-66d99547d9-44428\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:13:00.326174 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:00.326102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:13:00.467120 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:00.467092    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"]
Jul 16 21:13:00.469292 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:13:00.469258    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd8cdb6_e206_4e82_822d_09a73b958361.slice/crio-098a5fc8742739a09c57d94454a9554ad95644309789aca7ecc54a30759d5468 WatchSource:0}: Error finding container 098a5fc8742739a09c57d94454a9554ad95644309789aca7ecc54a30759d5468: Status 404 returned error can't find the container with id 098a5fc8742739a09c57d94454a9554ad95644309789aca7ecc54a30759d5468
Jul 16 21:13:01.220449 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:01.220357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428" event={"ID":"dbd8cdb6-e206-4e82-822d-09a73b958361","Type":"ContainerStarted","Data":"bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37"}
Jul 16 21:13:01.220449 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:01.220415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428" event={"ID":"dbd8cdb6-e206-4e82-822d-09a73b958361","Type":"ContainerStarted","Data":"098a5fc8742739a09c57d94454a9554ad95644309789aca7ecc54a30759d5468"}
Jul 16 21:13:01.220651 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:01.220469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:13:01.239106 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:01.239050    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428" podStartSLOduration=1.822932435 podStartE2EDuration="2.239034458s" totalImagesPullingTime="416.102023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:13:01.23749185 +0000 UTC m=+2388.340770838" watchObservedRunningTime="2026-07-16 21:13:01.239034458 +0000 UTC m=+2388.342313442"
Jul 16 21:13:07.230836 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:07.230805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:13:13.509506 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:13.509478    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 21:13:13.514545 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:13.514508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 21:13:31.207300 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.207266    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"]
Jul 16 21:13:31.210825 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.210808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.213636 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.213617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"default-dockercfg-9v6zv\""
Jul 16 21:13:31.214754 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.214737    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"kube-root-ca.crt\""
Jul 16 21:13:31.214846 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.214768    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"auth-test-model-69d8e7-kserve-self-signed-certs\""
Jul 16 21:13:31.214846 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.214768    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"openshift-service-ca.crt\""
Jul 16 21:13:31.229634 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.229604    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"]
Jul 16 21:13:31.328657 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.328629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-model-cache\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.328831 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.328664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbk95\" (UniqueName: \"kubernetes.io/projected/5e0cf852-b1ff-4b86-95c8-2963f5acce66-kube-api-access-dbk95\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.328831 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.328694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-home\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.328831 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.328793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-dshm\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.328831 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.328818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tmp-dir\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.328993 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.328844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tls-certs\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.429620 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.429585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-model-cache\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.429620 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.429628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbk95\" (UniqueName: \"kubernetes.io/projected/5e0cf852-b1ff-4b86-95c8-2963f5acce66-kube-api-access-dbk95\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.429874 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.429771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-home\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.429938 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.429874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-dshm\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.429938 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.429900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tmp-dir\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.430043 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.429935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tls-certs\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.430098 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.430058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-model-cache\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.430153 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.430139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-home\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.430210 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.430188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tmp-dir\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.432194 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.432166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-dshm\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.432482 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.432460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tls-certs\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.440749 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.440730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbk95\" (UniqueName: \"kubernetes.io/projected/5e0cf852-b1ff-4b86-95c8-2963f5acce66-kube-api-access-dbk95\") pod \"auth-test-model-69d8e7-kserve-6f74c674b7-hvblz\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") " pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.519358 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.519273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:31.669893 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:31.669870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"]
Jul 16 21:13:31.671115 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:13:31.671085    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e0cf852_b1ff_4b86_95c8_2963f5acce66.slice/crio-899d7f504ac46b0119674fb25291bf116e44d88a2f3c51bf924724674b08234d WatchSource:0}: Error finding container 899d7f504ac46b0119674fb25291bf116e44d88a2f3c51bf924724674b08234d: Status 404 returned error can't find the container with id 899d7f504ac46b0119674fb25291bf116e44d88a2f3c51bf924724674b08234d
Jul 16 21:13:32.367958 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:32.367918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz" event={"ID":"5e0cf852-b1ff-4b86-95c8-2963f5acce66","Type":"ContainerStarted","Data":"a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853"}
Jul 16 21:13:32.367958 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:32.367961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz" event={"ID":"5e0cf852-b1ff-4b86-95c8-2963f5acce66","Type":"ContainerStarted","Data":"899d7f504ac46b0119674fb25291bf116e44d88a2f3c51bf924724674b08234d"}
Jul 16 21:13:32.386372 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:32.386317    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz" podStartSLOduration=1.386300118 podStartE2EDuration="1.386300118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:13:32.385692425 +0000 UTC m=+2419.488971411" watchObservedRunningTime="2026-07-16 21:13:32.386300118 +0000 UTC m=+2419.489579103"
Jul 16 21:13:41.520273 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:41.520189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:41.520273 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:41.520222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:41.532622 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:41.532601    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:13:42.427267 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:13:42.427236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:15:26.951164 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:26.951118    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"]
Jul 16 21:15:41.970881 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:41.970829    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz" podUID="5e0cf852-b1ff-4b86-95c8-2963f5acce66" containerName="main" containerID="cri-o://a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853" gracePeriod=45
Jul 16 21:15:42.230507 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.230446    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:15:42.303130 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303094    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-dshm\") pod \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") "
Jul 16 21:15:42.303326 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-home\") pod \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") "
Jul 16 21:15:42.303326 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbk95\" (UniqueName: \"kubernetes.io/projected/5e0cf852-b1ff-4b86-95c8-2963f5acce66-kube-api-access-dbk95\") pod \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") "
Jul 16 21:15:42.303326 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-model-cache\") pod \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") "
Jul 16 21:15:42.303326 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303306    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tls-certs\") pod \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") "
Jul 16 21:15:42.303571 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303344    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tmp-dir\") pod \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\" (UID: \"5e0cf852-b1ff-4b86-95c8-2963f5acce66\") "
Jul 16 21:15:42.303571 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-home" (OuterVolumeSpecName: "home") pod "5e0cf852-b1ff-4b86-95c8-2963f5acce66" (UID: "5e0cf852-b1ff-4b86-95c8-2963f5acce66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:15:42.303571 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e0cf852-b1ff-4b86-95c8-2963f5acce66" (UID: "5e0cf852-b1ff-4b86-95c8-2963f5acce66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:15:42.303714 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303615    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:15:42.303714 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303631    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:15:42.303714 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.303646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e0cf852-b1ff-4b86-95c8-2963f5acce66" (UID: "5e0cf852-b1ff-4b86-95c8-2963f5acce66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:15:42.305468 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.305448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-dshm" (OuterVolumeSpecName: "dshm") pod "5e0cf852-b1ff-4b86-95c8-2963f5acce66" (UID: "5e0cf852-b1ff-4b86-95c8-2963f5acce66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:15:42.305559 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.305538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0cf852-b1ff-4b86-95c8-2963f5acce66-kube-api-access-dbk95" (OuterVolumeSpecName: "kube-api-access-dbk95") pod "5e0cf852-b1ff-4b86-95c8-2963f5acce66" (UID: "5e0cf852-b1ff-4b86-95c8-2963f5acce66"). InnerVolumeSpecName "kube-api-access-dbk95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:15:42.305631 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.305614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e0cf852-b1ff-4b86-95c8-2963f5acce66" (UID: "5e0cf852-b1ff-4b86-95c8-2963f5acce66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:15:42.404761 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.404729    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:15:42.404761 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.404755    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:15:42.404761 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.404764    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0cf852-b1ff-4b86-95c8-2963f5acce66-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:15:42.404994 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.404772    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbk95\" (UniqueName: \"kubernetes.io/projected/5e0cf852-b1ff-4b86-95c8-2963f5acce66-kube-api-access-dbk95\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:15:42.976714 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.976676    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e0cf852-b1ff-4b86-95c8-2963f5acce66" containerID="a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853" exitCode=0
Jul 16 21:15:42.976714 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.976716    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"
Jul 16 21:15:42.977294 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.976768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz" event={"ID":"5e0cf852-b1ff-4b86-95c8-2963f5acce66","Type":"ContainerDied","Data":"a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853"}
Jul 16 21:15:42.977294 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.976819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz" event={"ID":"5e0cf852-b1ff-4b86-95c8-2963f5acce66","Type":"ContainerDied","Data":"899d7f504ac46b0119674fb25291bf116e44d88a2f3c51bf924724674b08234d"}
Jul 16 21:15:42.977294 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.976843    2580 scope.go:117] "RemoveContainer" containerID="a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853"
Jul 16 21:15:42.986046 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.986031    2580 scope.go:117] "RemoveContainer" containerID="a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853"
Jul 16 21:15:42.986304 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:15:42.986283    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853\": container with ID starting with a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853 not found: ID does not exist" containerID="a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853"
Jul 16 21:15:42.986346 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.986312    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853"} err="failed to get container status \"a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853\": rpc error: code = NotFound desc = could not find container \"a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853\": container with ID starting with a658f920336a506de7ab0eace8b5cd670eff0d39195013ed597a244ad6933853 not found: ID does not exist"
Jul 16 21:15:42.998252 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:42.998226    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"]
Jul 16 21:15:43.005675 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:43.005656    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/auth-test-model-69d8e7-kserve-6f74c674b7-hvblz"]
Jul 16 21:15:43.504844 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:15:43.504801    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e0cf852-b1ff-4b86-95c8-2963f5acce66" path="/var/lib/kubelet/pods/5e0cf852-b1ff-4b86-95c8-2963f5acce66/volumes"
Jul 16 21:16:02.410023 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.409993    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"]
Jul 16 21:16:02.410407 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.410364    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e0cf852-b1ff-4b86-95c8-2963f5acce66" containerName="main"
Jul 16 21:16:02.410407 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.410375    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e0cf852-b1ff-4b86-95c8-2963f5acce66" containerName="main"
Jul 16 21:16:02.410486 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.410428    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e0cf852-b1ff-4b86-95c8-2963f5acce66" containerName="main"
Jul 16 21:16:02.413973 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.413953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.416906 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.416885    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"sub-test-model-69d8e7-kserve-self-signed-certs\""
Jul 16 21:16:02.416996 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.416905    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"openshift-service-ca.crt\""
Jul 16 21:16:02.419346 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.419323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"kube-root-ca.crt\""
Jul 16 21:16:02.419967 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.419944    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"default-dockercfg-9v6zv\""
Jul 16 21:16:02.429758 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.429728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"]
Jul 16 21:16:02.464274 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.464237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-model-cache\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.464463 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.464281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tls-certs\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.464463 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.464341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-dshm\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.464463 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.464363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tmp-dir\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.464463 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.464388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-home\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.464463 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.464447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfzx\" (UniqueName: \"kubernetes.io/projected/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-kube-api-access-swfzx\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.565747 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.565714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfzx\" (UniqueName: \"kubernetes.io/projected/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-kube-api-access-swfzx\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.565929 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.565898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-model-cache\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.565989 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.565937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tls-certs\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.565989 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.565981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-dshm\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.566094 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.565998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tmp-dir\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.566094 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.566017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-home\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.566333 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.566313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-model-cache\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.566396 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.566358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tmp-dir\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.566485 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.566469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-home\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.568285 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.568263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-dshm\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.568492 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.568475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tls-certs\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.575551 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.575529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfzx\" (UniqueName: \"kubernetes.io/projected/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-kube-api-access-swfzx\") pod \"sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") " pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.722648 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.722547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:02.861757 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:02.861731    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"]
Jul 16 21:16:02.864328 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:16:02.864298    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2b6d5d9_6c28_4d1d_a768_a78a66187e23.slice/crio-c88acacaf8231441c3afda2005526679f9c134dacd4ab936ea59a2c9c4fb34ef WatchSource:0}: Error finding container c88acacaf8231441c3afda2005526679f9c134dacd4ab936ea59a2c9c4fb34ef: Status 404 returned error can't find the container with id c88acacaf8231441c3afda2005526679f9c134dacd4ab936ea59a2c9c4fb34ef
Jul 16 21:16:03.070884 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:03.070803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx" event={"ID":"e2b6d5d9-6c28-4d1d-a768-a78a66187e23","Type":"ContainerStarted","Data":"51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811"}
Jul 16 21:16:03.070884 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:03.070842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx" event={"ID":"e2b6d5d9-6c28-4d1d-a768-a78a66187e23","Type":"ContainerStarted","Data":"c88acacaf8231441c3afda2005526679f9c134dacd4ab936ea59a2c9c4fb34ef"}
Jul 16 21:16:03.091275 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:03.091221    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx" podStartSLOduration=1.091204956 podStartE2EDuration="1.091204956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:16:03.088794429 +0000 UTC m=+2570.192073418" watchObservedRunningTime="2026-07-16 21:16:03.091204956 +0000 UTC m=+2570.194483938"
Jul 16 21:16:12.723126 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:12.723082    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:12.723126 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:12.723131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:12.736395 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:12.736369    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:13.131386 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:13.131300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:16:58.442736 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:16:58.442650    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"]
Jul 16 21:17:13.462912 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.462832    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx" podUID="e2b6d5d9-6c28-4d1d-a768-a78a66187e23" containerName="main" containerID="cri-o://51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811" gracePeriod=45
Jul 16 21:17:13.713260 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.713186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:17:13.805810 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.805771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-model-cache\") pod \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") "
Jul 16 21:17:13.806016 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.805844    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-home\") pod \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") "
Jul 16 21:17:13.806016 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.805880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tls-certs\") pod \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") "
Jul 16 21:17:13.806016 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.805921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tmp-dir\") pod \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") "
Jul 16 21:17:13.806016 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.805947    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swfzx\" (UniqueName: \"kubernetes.io/projected/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-kube-api-access-swfzx\") pod \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") "
Jul 16 21:17:13.806016 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.805977    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-dshm\") pod \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\" (UID: \"e2b6d5d9-6c28-4d1d-a768-a78a66187e23\") "
Jul 16 21:17:13.806279 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.806122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-home" (OuterVolumeSpecName: "home") pod "e2b6d5d9-6c28-4d1d-a768-a78a66187e23" (UID: "e2b6d5d9-6c28-4d1d-a768-a78a66187e23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:17:13.806279 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.806141    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2b6d5d9-6c28-4d1d-a768-a78a66187e23" (UID: "e2b6d5d9-6c28-4d1d-a768-a78a66187e23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:17:13.806379 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.806291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2b6d5d9-6c28-4d1d-a768-a78a66187e23" (UID: "e2b6d5d9-6c28-4d1d-a768-a78a66187e23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:17:13.806379 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.806318    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:17:13.806379 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.806338    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:17:13.807988 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.807963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-dshm" (OuterVolumeSpecName: "dshm") pod "e2b6d5d9-6c28-4d1d-a768-a78a66187e23" (UID: "e2b6d5d9-6c28-4d1d-a768-a78a66187e23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:17:13.808089 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.808045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-kube-api-access-swfzx" (OuterVolumeSpecName: "kube-api-access-swfzx") pod "e2b6d5d9-6c28-4d1d-a768-a78a66187e23" (UID: "e2b6d5d9-6c28-4d1d-a768-a78a66187e23"). InnerVolumeSpecName "kube-api-access-swfzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:17:13.808089 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.808063    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2b6d5d9-6c28-4d1d-a768-a78a66187e23" (UID: "e2b6d5d9-6c28-4d1d-a768-a78a66187e23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:17:13.907199 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.907155    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:17:13.907199 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.907191    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:17:13.907199 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.907202    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swfzx\" (UniqueName: \"kubernetes.io/projected/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-kube-api-access-swfzx\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:17:13.907442 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:13.907213    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b6d5d9-6c28-4d1d-a768-a78a66187e23-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:17:14.399022 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.398989    2580 generic.go:356] "Generic (PLEG): container finished" podID="e2b6d5d9-6c28-4d1d-a768-a78a66187e23" containerID="51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811" exitCode=0
Jul 16 21:17:14.399238 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.399028    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"
Jul 16 21:17:14.399238 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.399082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx" event={"ID":"e2b6d5d9-6c28-4d1d-a768-a78a66187e23","Type":"ContainerDied","Data":"51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811"}
Jul 16 21:17:14.399238 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.399128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx" event={"ID":"e2b6d5d9-6c28-4d1d-a768-a78a66187e23","Type":"ContainerDied","Data":"c88acacaf8231441c3afda2005526679f9c134dacd4ab936ea59a2c9c4fb34ef"}
Jul 16 21:17:14.399238 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.399151    2580 scope.go:117] "RemoveContainer" containerID="51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811"
Jul 16 21:17:14.408493 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.408470    2580 scope.go:117] "RemoveContainer" containerID="51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811"
Jul 16 21:17:14.408803 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:17:14.408784    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811\": container with ID starting with 51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811 not found: ID does not exist" containerID="51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811"
Jul 16 21:17:14.408852 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.408812    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811"} err="failed to get container status \"51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811\": rpc error: code = NotFound desc = could not find container \"51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811\": container with ID starting with 51f02c4679f55c097a81c3d42a28d2c7229e675921f11c24e8b4b412f0228811 not found: ID does not exist"
Jul 16 21:17:14.421689 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.421660    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"]
Jul 16 21:17:14.425471 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:14.425440    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/sub-test-model-69d8e7-kserve-646cfd5b7c-8cqwx"]
Jul 16 21:17:15.498365 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:17:15.498330    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b6d5d9-6c28-4d1d-a768-a78a66187e23" path="/var/lib/kubelet/pods/e2b6d5d9-6c28-4d1d-a768-a78a66187e23/volumes"
Jul 16 21:18:13.537311 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:13.537274    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 21:18:13.542815 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:13.542791    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qszx_a839f97a-d4f3-4093-8c47-1ab868fce6f9/ovn-acl-logging/0.log"
Jul 16 21:18:43.483507 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.483475    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7154-140581940534704"]
Jul 16 21:18:43.483981 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.483835    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b6d5d9-6c28-4d1d-a768-a78a66187e23" containerName="main"
Jul 16 21:18:43.483981 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.483848    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b6d5d9-6c28-4d1d-a768-a78a66187e23" containerName="main"
Jul 16 21:18:43.483981 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.483935    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b6d5d9-6c28-4d1d-a768-a78a66187e23" containerName="main"
Jul 16 21:18:43.486897 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.486881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581940534704"
Jul 16 21:18:43.490179 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.490153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fq898\""
Jul 16 21:18:43.500052 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.500026    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581940534704"]
Jul 16 21:18:43.621090 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.621056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vvm\" (UniqueName: \"kubernetes.io/projected/72fe89ea-bd97-47ae-8335-04c7fae1d75d-kube-api-access-x8vvm\") pod \"test-curl-7154-140581940534704\" (UID: \"72fe89ea-bd97-47ae-8335-04c7fae1d75d\") " pod="opendatahub/test-curl-7154-140581940534704"
Jul 16 21:18:43.722571 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.722538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vvm\" (UniqueName: \"kubernetes.io/projected/72fe89ea-bd97-47ae-8335-04c7fae1d75d-kube-api-access-x8vvm\") pod \"test-curl-7154-140581940534704\" (UID: \"72fe89ea-bd97-47ae-8335-04c7fae1d75d\") " pod="opendatahub/test-curl-7154-140581940534704"
Jul 16 21:18:43.732041 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.732014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vvm\" (UniqueName: \"kubernetes.io/projected/72fe89ea-bd97-47ae-8335-04c7fae1d75d-kube-api-access-x8vvm\") pod \"test-curl-7154-140581940534704\" (UID: \"72fe89ea-bd97-47ae-8335-04c7fae1d75d\") " pod="opendatahub/test-curl-7154-140581940534704"
Jul 16 21:18:43.797266 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.797186    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581940534704"
Jul 16 21:18:43.926935 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.926912    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581940534704"]
Jul 16 21:18:43.929039 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:18:43.929009    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72fe89ea_bd97_47ae_8335_04c7fae1d75d.slice/crio-710692716fb4fd06bc0e6893050c1f6929faa8eff41e40e444aa989de8d84eab WatchSource:0}: Error finding container 710692716fb4fd06bc0e6893050c1f6929faa8eff41e40e444aa989de8d84eab: Status 404 returned error can't find the container with id 710692716fb4fd06bc0e6893050c1f6929faa8eff41e40e444aa989de8d84eab
Jul 16 21:18:43.930139 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:43.930123    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:18:44.817131 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:44.817098    2580 generic.go:356] "Generic (PLEG): container finished" podID="72fe89ea-bd97-47ae-8335-04c7fae1d75d" containerID="96818d37f3ee4c68654b278bd6a9373c6dad43328dafb405316ef1cd994771e3" exitCode=0
Jul 16 21:18:44.817586 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:44.817159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581940534704" event={"ID":"72fe89ea-bd97-47ae-8335-04c7fae1d75d","Type":"ContainerDied","Data":"96818d37f3ee4c68654b278bd6a9373c6dad43328dafb405316ef1cd994771e3"}
Jul 16 21:18:44.817586 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:44.817186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581940534704" event={"ID":"72fe89ea-bd97-47ae-8335-04c7fae1d75d","Type":"ContainerStarted","Data":"710692716fb4fd06bc0e6893050c1f6929faa8eff41e40e444aa989de8d84eab"}
Jul 16 21:18:45.958101 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:45.958076    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581940534704"
Jul 16 21:18:45.995355 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:45.995325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7154-140581940534704_72fe89ea-bd97-47ae-8335-04c7fae1d75d/test-curl-7154-140581940534704/0.log"
Jul 16 21:18:46.020466 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.020432    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7154-140581940534704"]
Jul 16 21:18:46.024109 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.024085    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7154-140581940534704"]
Jul 16 21:18:46.044590 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.044570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8vvm\" (UniqueName: \"kubernetes.io/projected/72fe89ea-bd97-47ae-8335-04c7fae1d75d-kube-api-access-x8vvm\") pod \"72fe89ea-bd97-47ae-8335-04c7fae1d75d\" (UID: \"72fe89ea-bd97-47ae-8335-04c7fae1d75d\") "
Jul 16 21:18:46.046616 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.046587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fe89ea-bd97-47ae-8335-04c7fae1d75d-kube-api-access-x8vvm" (OuterVolumeSpecName: "kube-api-access-x8vvm") pod "72fe89ea-bd97-47ae-8335-04c7fae1d75d" (UID: "72fe89ea-bd97-47ae-8335-04c7fae1d75d"). InnerVolumeSpecName "kube-api-access-x8vvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:18:46.145409 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.145374    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8vvm\" (UniqueName: \"kubernetes.io/projected/72fe89ea-bd97-47ae-8335-04c7fae1d75d-kube-api-access-x8vvm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:18:46.151874 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.151846    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7154-140581940692528"]
Jul 16 21:18:46.152244 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.152221    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72fe89ea-bd97-47ae-8335-04c7fae1d75d" containerName="test-curl-7154-140581940534704"
Jul 16 21:18:46.152292 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.152246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72fe89ea-bd97-47ae-8335-04c7fae1d75d" containerName="test-curl-7154-140581940534704"
Jul 16 21:18:46.152327 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.152303    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72fe89ea-bd97-47ae-8335-04c7fae1d75d" containerName="test-curl-7154-140581940534704"
Jul 16 21:18:46.155193 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.155179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581940692528"
Jul 16 21:18:46.161826 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.161785    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581940692528"]
Jul 16 21:18:46.246537 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.246476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqmc\" (UniqueName: \"kubernetes.io/projected/d129d831-a85b-4846-bcfe-d02d4697137d-kube-api-access-ngqmc\") pod \"test-curl-7154-140581940692528\" (UID: \"d129d831-a85b-4846-bcfe-d02d4697137d\") " pod="opendatahub/test-curl-7154-140581940692528"
Jul 16 21:18:46.347766 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.347734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqmc\" (UniqueName: \"kubernetes.io/projected/d129d831-a85b-4846-bcfe-d02d4697137d-kube-api-access-ngqmc\") pod \"test-curl-7154-140581940692528\" (UID: \"d129d831-a85b-4846-bcfe-d02d4697137d\") " pod="opendatahub/test-curl-7154-140581940692528"
Jul 16 21:18:46.356588 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.356563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqmc\" (UniqueName: \"kubernetes.io/projected/d129d831-a85b-4846-bcfe-d02d4697137d-kube-api-access-ngqmc\") pod \"test-curl-7154-140581940692528\" (UID: \"d129d831-a85b-4846-bcfe-d02d4697137d\") " pod="opendatahub/test-curl-7154-140581940692528"
Jul 16 21:18:46.464833 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.464758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581940692528"
Jul 16 21:18:46.590935 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.590913    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581940692528"]
Jul 16 21:18:46.593308 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:18:46.593283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd129d831_a85b_4846_bcfe_d02d4697137d.slice/crio-872f5186d7bbb96b7ada2729acb951f25c3d2f91e7f2e4cb218c1be544ca943c WatchSource:0}: Error finding container 872f5186d7bbb96b7ada2729acb951f25c3d2f91e7f2e4cb218c1be544ca943c: Status 404 returned error can't find the container with id 872f5186d7bbb96b7ada2729acb951f25c3d2f91e7f2e4cb218c1be544ca943c
Jul 16 21:18:46.834488 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.834396    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581940534704"
Jul 16 21:18:46.834488 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.834417    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710692716fb4fd06bc0e6893050c1f6929faa8eff41e40e444aa989de8d84eab"
Jul 16 21:18:46.835567 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:46.835541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581940692528" event={"ID":"d129d831-a85b-4846-bcfe-d02d4697137d","Type":"ContainerStarted","Data":"872f5186d7bbb96b7ada2729acb951f25c3d2f91e7f2e4cb218c1be544ca943c"}
Jul 16 21:18:46.845312 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:18:46.845281    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7154-140581940534704\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="72fe89ea-bd97-47ae-8335-04c7fae1d75d" pod="opendatahub/test-curl-7154-140581940534704"
Jul 16 21:18:47.499122 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:47.499086    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72fe89ea-bd97-47ae-8335-04c7fae1d75d" path="/var/lib/kubelet/pods/72fe89ea-bd97-47ae-8335-04c7fae1d75d/volumes"
Jul 16 21:18:47.841822 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:47.841735    2580 generic.go:356] "Generic (PLEG): container finished" podID="d129d831-a85b-4846-bcfe-d02d4697137d" containerID="514f620a400c089fbe7285d8d9fecad232191793f283d6f344d3911616fab353" exitCode=0
Jul 16 21:18:47.841972 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:47.841825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581940692528" event={"ID":"d129d831-a85b-4846-bcfe-d02d4697137d","Type":"ContainerDied","Data":"514f620a400c089fbe7285d8d9fecad232191793f283d6f344d3911616fab353"}
Jul 16 21:18:48.985068 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:48.985044    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581940692528"
Jul 16 21:18:49.010508 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.010487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7154-140581940692528_d129d831-a85b-4846-bcfe-d02d4697137d/test-curl-7154-140581940692528/0.log"
Jul 16 21:18:49.035887 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.035856    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7154-140581940692528"]
Jul 16 21:18:49.041438 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.041414    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7154-140581940692528"]
Jul 16 21:18:49.072123 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.072097    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngqmc\" (UniqueName: \"kubernetes.io/projected/d129d831-a85b-4846-bcfe-d02d4697137d-kube-api-access-ngqmc\") pod \"d129d831-a85b-4846-bcfe-d02d4697137d\" (UID: \"d129d831-a85b-4846-bcfe-d02d4697137d\") "
Jul 16 21:18:49.074111 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.074083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d129d831-a85b-4846-bcfe-d02d4697137d-kube-api-access-ngqmc" (OuterVolumeSpecName: "kube-api-access-ngqmc") pod "d129d831-a85b-4846-bcfe-d02d4697137d" (UID: "d129d831-a85b-4846-bcfe-d02d4697137d"). InnerVolumeSpecName "kube-api-access-ngqmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:18:49.173468 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.173430    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqmc\" (UniqueName: \"kubernetes.io/projected/d129d831-a85b-4846-bcfe-d02d4697137d-kube-api-access-ngqmc\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:18:49.210159 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.210125    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:49.210487 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.210474    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d129d831-a85b-4846-bcfe-d02d4697137d" containerName="test-curl-7154-140581940692528"
Jul 16 21:18:49.210554 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.210489    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d129d831-a85b-4846-bcfe-d02d4697137d" containerName="test-curl-7154-140581940692528"
Jul 16 21:18:49.210595 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.210579    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d129d831-a85b-4846-bcfe-d02d4697137d" containerName="test-curl-7154-140581940692528"
Jul 16 21:18:49.213569 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.213553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:49.220594 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.220387    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:49.375390 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.375356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9cg4\" (UniqueName: \"kubernetes.io/projected/0c314045-795f-4da2-9961-bb8bfe853dd3-kube-api-access-f9cg4\") pod \"test-curl-7154-140581941126032\" (UID: \"0c314045-795f-4da2-9961-bb8bfe853dd3\") " pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:49.476245 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.476155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cg4\" (UniqueName: \"kubernetes.io/projected/0c314045-795f-4da2-9961-bb8bfe853dd3-kube-api-access-f9cg4\") pod \"test-curl-7154-140581941126032\" (UID: \"0c314045-795f-4da2-9961-bb8bfe853dd3\") " pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:49.487567 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.487536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cg4\" (UniqueName: \"kubernetes.io/projected/0c314045-795f-4da2-9961-bb8bfe853dd3-kube-api-access-f9cg4\") pod \"test-curl-7154-140581941126032\" (UID: \"0c314045-795f-4da2-9961-bb8bfe853dd3\") " pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:49.499267 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.499239    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d129d831-a85b-4846-bcfe-d02d4697137d" path="/var/lib/kubelet/pods/d129d831-a85b-4846-bcfe-d02d4697137d/volumes"
Jul 16 21:18:49.523738 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.523712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:49.657760 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.657729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:49.660751 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:18:49.660723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c314045_795f_4da2_9961_bb8bfe853dd3.slice/crio-12035d26ac297a214f5b8cf7e9dbe28ad8645540dc00fd3c10856514e89c42a3 WatchSource:0}: Error finding container 12035d26ac297a214f5b8cf7e9dbe28ad8645540dc00fd3c10856514e89c42a3: Status 404 returned error can't find the container with id 12035d26ac297a214f5b8cf7e9dbe28ad8645540dc00fd3c10856514e89c42a3
Jul 16 21:18:49.851661 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.851568    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581940692528"
Jul 16 21:18:49.851661 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.851608    2580 scope.go:117] "RemoveContainer" containerID="514f620a400c089fbe7285d8d9fecad232191793f283d6f344d3911616fab353"
Jul 16 21:18:49.852959 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:49.852918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581941126032" event={"ID":"0c314045-795f-4da2-9961-bb8bfe853dd3","Type":"ContainerStarted","Data":"12035d26ac297a214f5b8cf7e9dbe28ad8645540dc00fd3c10856514e89c42a3"}
Jul 16 21:18:50.860335 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:50.860303    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c314045-795f-4da2-9961-bb8bfe853dd3" containerID="127da0144ac7450fec16a65c4e3aa962022b303c29268cb11c2bde3e0264bc24" exitCode=0
Jul 16 21:18:50.860768 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:50.860386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581941126032" event={"ID":"0c314045-795f-4da2-9961-bb8bfe853dd3","Type":"ContainerDied","Data":"127da0144ac7450fec16a65c4e3aa962022b303c29268cb11c2bde3e0264bc24"}
Jul 16 21:18:51.999275 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:51.999246    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:52.022582 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.022549    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7154-140581941126032_0c314045-795f-4da2-9961-bb8bfe853dd3/test-curl-7154-140581941126032/0.log"
Jul 16 21:18:52.046745 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.046701    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:52.050759 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.050732    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:52.102342 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.102311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9cg4\" (UniqueName: \"kubernetes.io/projected/0c314045-795f-4da2-9961-bb8bfe853dd3-kube-api-access-f9cg4\") pod \"0c314045-795f-4da2-9961-bb8bfe853dd3\" (UID: \"0c314045-795f-4da2-9961-bb8bfe853dd3\") "
Jul 16 21:18:52.104350 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.104318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c314045-795f-4da2-9961-bb8bfe853dd3-kube-api-access-f9cg4" (OuterVolumeSpecName: "kube-api-access-f9cg4") pod "0c314045-795f-4da2-9961-bb8bfe853dd3" (UID: "0c314045-795f-4da2-9961-bb8bfe853dd3"). InnerVolumeSpecName "kube-api-access-f9cg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:18:52.168980 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.168949    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:52.169347 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.169335    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c314045-795f-4da2-9961-bb8bfe853dd3" containerName="test-curl-7154-140581941126032"
Jul 16 21:18:52.169396 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.169350    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c314045-795f-4da2-9961-bb8bfe853dd3" containerName="test-curl-7154-140581941126032"
Jul 16 21:18:52.169434 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.169422    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c314045-795f-4da2-9961-bb8bfe853dd3" containerName="test-curl-7154-140581941126032"
Jul 16 21:18:52.177007 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.176982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:52.179164 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.179137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:52.203454 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.203416    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9cg4\" (UniqueName: \"kubernetes.io/projected/0c314045-795f-4da2-9961-bb8bfe853dd3-kube-api-access-f9cg4\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:18:52.304675 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.304640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgs62\" (UniqueName: \"kubernetes.io/projected/e4e87848-142e-4716-97f0-27dee4db6aa7-kube-api-access-jgs62\") pod \"test-curl-7154-140581941126032\" (UID: \"e4e87848-142e-4716-97f0-27dee4db6aa7\") " pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:52.405542 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.405474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgs62\" (UniqueName: \"kubernetes.io/projected/e4e87848-142e-4716-97f0-27dee4db6aa7-kube-api-access-jgs62\") pod \"test-curl-7154-140581941126032\" (UID: \"e4e87848-142e-4716-97f0-27dee4db6aa7\") " pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:52.414933 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.414910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgs62\" (UniqueName: \"kubernetes.io/projected/e4e87848-142e-4716-97f0-27dee4db6aa7-kube-api-access-jgs62\") pod \"test-curl-7154-140581941126032\" (UID: \"e4e87848-142e-4716-97f0-27dee4db6aa7\") " pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:52.487005 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.486910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:52.621802 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.621747    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:52.625408 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:18:52.625378    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4e87848_142e_4716_97f0_27dee4db6aa7.slice/crio-6c5219e037a3d0e0c880b4ae288303643a50fe445bd2af20212d934a80d913e5 WatchSource:0}: Error finding container 6c5219e037a3d0e0c880b4ae288303643a50fe445bd2af20212d934a80d913e5: Status 404 returned error can't find the container with id 6c5219e037a3d0e0c880b4ae288303643a50fe445bd2af20212d934a80d913e5
Jul 16 21:18:52.873644 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.873558    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12035d26ac297a214f5b8cf7e9dbe28ad8645540dc00fd3c10856514e89c42a3"
Jul 16 21:18:52.873644 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.873555    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:52.874718 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.874681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581941126032" event={"ID":"e4e87848-142e-4716-97f0-27dee4db6aa7","Type":"ContainerStarted","Data":"6c5219e037a3d0e0c880b4ae288303643a50fe445bd2af20212d934a80d913e5"}
Jul 16 21:18:52.885951 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:52.885923    2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7154-140581941126032" oldPodUID="0c314045-795f-4da2-9961-bb8bfe853dd3" podUID="e4e87848-142e-4716-97f0-27dee4db6aa7"
Jul 16 21:18:53.498919 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:53.498877    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c314045-795f-4da2-9961-bb8bfe853dd3" path="/var/lib/kubelet/pods/0c314045-795f-4da2-9961-bb8bfe853dd3/volumes"
Jul 16 21:18:53.880997 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:53.880914    2580 generic.go:356] "Generic (PLEG): container finished" podID="e4e87848-142e-4716-97f0-27dee4db6aa7" containerID="bddd82f93595f509a7d6077797c58d9aa917c43ce6206c76e4c6b72f54d4ea34" exitCode=0
Jul 16 21:18:53.880997 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:53.880958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581941126032" event={"ID":"e4e87848-142e-4716-97f0-27dee4db6aa7","Type":"ContainerDied","Data":"bddd82f93595f509a7d6077797c58d9aa917c43ce6206c76e4c6b72f54d4ea34"}
Jul 16 21:18:55.023919 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.023894    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:55.047157 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.047127    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7154-140581941126032_e4e87848-142e-4716-97f0-27dee4db6aa7/test-curl-7154-140581941126032/0.log"
Jul 16 21:18:55.075646 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.075605    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:55.084348 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.084310    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7154-140581941126032"]
Jul 16 21:18:55.131550 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.131498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgs62\" (UniqueName: \"kubernetes.io/projected/e4e87848-142e-4716-97f0-27dee4db6aa7-kube-api-access-jgs62\") pod \"e4e87848-142e-4716-97f0-27dee4db6aa7\" (UID: \"e4e87848-142e-4716-97f0-27dee4db6aa7\") "
Jul 16 21:18:55.133618 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.133597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e87848-142e-4716-97f0-27dee4db6aa7-kube-api-access-jgs62" (OuterVolumeSpecName: "kube-api-access-jgs62") pod "e4e87848-142e-4716-97f0-27dee4db6aa7" (UID: "e4e87848-142e-4716-97f0-27dee4db6aa7"). InnerVolumeSpecName "kube-api-access-jgs62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:18:55.203447 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.203409    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7154-140581941127040"]
Jul 16 21:18:55.203872 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.203853    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e87848-142e-4716-97f0-27dee4db6aa7" containerName="test-curl-7154-140581941126032"
Jul 16 21:18:55.203948 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.203874    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e87848-142e-4716-97f0-27dee4db6aa7" containerName="test-curl-7154-140581941126032"
Jul 16 21:18:55.204013 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.203951    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e87848-142e-4716-97f0-27dee4db6aa7" containerName="test-curl-7154-140581941126032"
Jul 16 21:18:55.206906 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.206888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941127040"
Jul 16 21:18:55.212958 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.212936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581941127040"]
Jul 16 21:18:55.232902 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.232878    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgs62\" (UniqueName: \"kubernetes.io/projected/e4e87848-142e-4716-97f0-27dee4db6aa7-kube-api-access-jgs62\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:18:55.333454 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.333425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rf5\" (UniqueName: \"kubernetes.io/projected/cba96290-bab9-4984-9e15-729ab49b047f-kube-api-access-z4rf5\") pod \"test-curl-7154-140581941127040\" (UID: \"cba96290-bab9-4984-9e15-729ab49b047f\") " pod="opendatahub/test-curl-7154-140581941127040"
Jul 16 21:18:55.434453 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.434384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rf5\" (UniqueName: \"kubernetes.io/projected/cba96290-bab9-4984-9e15-729ab49b047f-kube-api-access-z4rf5\") pod \"test-curl-7154-140581941127040\" (UID: \"cba96290-bab9-4984-9e15-729ab49b047f\") " pod="opendatahub/test-curl-7154-140581941127040"
Jul 16 21:18:55.443747 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.443725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rf5\" (UniqueName: \"kubernetes.io/projected/cba96290-bab9-4984-9e15-729ab49b047f-kube-api-access-z4rf5\") pod \"test-curl-7154-140581941127040\" (UID: \"cba96290-bab9-4984-9e15-729ab49b047f\") " pod="opendatahub/test-curl-7154-140581941127040"
Jul 16 21:18:55.498860 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.498830    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e87848-142e-4716-97f0-27dee4db6aa7" path="/var/lib/kubelet/pods/e4e87848-142e-4716-97f0-27dee4db6aa7/volumes"
Jul 16 21:18:55.516855 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.516832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941127040"
Jul 16 21:18:55.650398 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.650368    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581941127040"]
Jul 16 21:18:55.652573 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:18:55.652553    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcba96290_bab9_4984_9e15_729ab49b047f.slice/crio-cc24bedf111ac7997c796f5ab03d863e5892eb30398b16430d11392f873c3f8b WatchSource:0}: Error finding container cc24bedf111ac7997c796f5ab03d863e5892eb30398b16430d11392f873c3f8b: Status 404 returned error can't find the container with id cc24bedf111ac7997c796f5ab03d863e5892eb30398b16430d11392f873c3f8b
Jul 16 21:18:55.891086 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.891054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581941127040" event={"ID":"cba96290-bab9-4984-9e15-729ab49b047f","Type":"ContainerStarted","Data":"cc24bedf111ac7997c796f5ab03d863e5892eb30398b16430d11392f873c3f8b"}
Jul 16 21:18:55.892231 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.892212    2580 scope.go:117] "RemoveContainer" containerID="bddd82f93595f509a7d6077797c58d9aa917c43ce6206c76e4c6b72f54d4ea34"
Jul 16 21:18:55.892323 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:55.892235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941126032"
Jul 16 21:18:56.898570 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:56.898529    2580 generic.go:356] "Generic (PLEG): container finished" podID="cba96290-bab9-4984-9e15-729ab49b047f" containerID="bf9340590fe7b8e0176dcd86479417b9ad14a87d75f6d28858bbd05e8ef31a33" exitCode=0
Jul 16 21:18:56.898979 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:56.898590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581941127040" event={"ID":"cba96290-bab9-4984-9e15-729ab49b047f","Type":"ContainerDied","Data":"bf9340590fe7b8e0176dcd86479417b9ad14a87d75f6d28858bbd05e8ef31a33"}
Jul 16 21:18:58.040615 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.040591    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941127040"
Jul 16 21:18:58.065607 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.065581    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7154-140581941127040_cba96290-bab9-4984-9e15-729ab49b047f/test-curl-7154-140581941127040/0.log"
Jul 16 21:18:58.095815 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.095781    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7154-140581941127040"]
Jul 16 21:18:58.100127 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.100101    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7154-140581941127040"]
Jul 16 21:18:58.155466 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.155384    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4rf5\" (UniqueName: \"kubernetes.io/projected/cba96290-bab9-4984-9e15-729ab49b047f-kube-api-access-z4rf5\") pod \"cba96290-bab9-4984-9e15-729ab49b047f\" (UID: \"cba96290-bab9-4984-9e15-729ab49b047f\") "
Jul 16 21:18:58.157474 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.157444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba96290-bab9-4984-9e15-729ab49b047f-kube-api-access-z4rf5" (OuterVolumeSpecName: "kube-api-access-z4rf5") pod "cba96290-bab9-4984-9e15-729ab49b047f" (UID: "cba96290-bab9-4984-9e15-729ab49b047f"). InnerVolumeSpecName "kube-api-access-z4rf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:18:58.256745 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.256715    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4rf5\" (UniqueName: \"kubernetes.io/projected/cba96290-bab9-4984-9e15-729ab49b047f-kube-api-access-z4rf5\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:18:58.909486 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.909451    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941127040"
Jul 16 21:18:58.909486 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:58.909477    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc24bedf111ac7997c796f5ab03d863e5892eb30398b16430d11392f873c3f8b"
Jul 16 21:18:58.920040 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:18:58.920009    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7154-140581941127040\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="cba96290-bab9-4984-9e15-729ab49b047f" pod="opendatahub/test-curl-7154-140581941127040"
Jul 16 21:18:59.499965 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:18:59.499922    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba96290-bab9-4984-9e15-729ab49b047f" path="/var/lib/kubelet/pods/cba96290-bab9-4984-9e15-729ab49b047f/volumes"
Jul 16 21:19:11.525994 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.525960    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"]
Jul 16 21:19:11.526391 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.526166    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk" podUID="dc60fc00-d091-462b-b897-f1b1b30d5361" containerName="maas-api" containerID="cri-o://18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877" gracePeriod=30
Jul 16 21:19:11.781880 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.781808    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:19:11.873996 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.873959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60fc00-d091-462b-b897-f1b1b30d5361-openshift-service-ca\") pod \"dc60fc00-d091-462b-b897-f1b1b30d5361\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") "
Jul 16 21:19:11.874173 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.874044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxgms\" (UniqueName: \"kubernetes.io/projected/dc60fc00-d091-462b-b897-f1b1b30d5361-kube-api-access-wxgms\") pod \"dc60fc00-d091-462b-b897-f1b1b30d5361\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") "
Jul 16 21:19:11.874173 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.874075    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls\") pod \"dc60fc00-d091-462b-b897-f1b1b30d5361\" (UID: \"dc60fc00-d091-462b-b897-f1b1b30d5361\") "
Jul 16 21:19:11.874366 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.874338    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc60fc00-d091-462b-b897-f1b1b30d5361-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc60fc00-d091-462b-b897-f1b1b30d5361" (UID: "dc60fc00-d091-462b-b897-f1b1b30d5361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:19:11.876170 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.876137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc60fc00-d091-462b-b897-f1b1b30d5361" (UID: "dc60fc00-d091-462b-b897-f1b1b30d5361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:19:11.876285 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.876168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc60fc00-d091-462b-b897-f1b1b30d5361-kube-api-access-wxgms" (OuterVolumeSpecName: "kube-api-access-wxgms") pod "dc60fc00-d091-462b-b897-f1b1b30d5361" (UID: "dc60fc00-d091-462b-b897-f1b1b30d5361"). InnerVolumeSpecName "kube-api-access-wxgms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:19:11.972708 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.972668    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc60fc00-d091-462b-b897-f1b1b30d5361" containerID="18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877" exitCode=0
Jul 16 21:19:11.972708 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.972714    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"
Jul 16 21:19:11.972915 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.972760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk" event={"ID":"dc60fc00-d091-462b-b897-f1b1b30d5361","Type":"ContainerDied","Data":"18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877"}
Jul 16 21:19:11.972915 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.972810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk" event={"ID":"dc60fc00-d091-462b-b897-f1b1b30d5361","Type":"ContainerDied","Data":"efa4bbda8a5b092689f1b6441ddf4d013170530d6dd98ef82d2d820614972006"}
Jul 16 21:19:11.972915 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.972834    2580 scope.go:117] "RemoveContainer" containerID="18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877"
Jul 16 21:19:11.974830 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.974804    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60fc00-d091-462b-b897-f1b1b30d5361-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:19:11.974830 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.974830    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxgms\" (UniqueName: \"kubernetes.io/projected/dc60fc00-d091-462b-b897-f1b1b30d5361-kube-api-access-wxgms\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:19:11.974945 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.974845    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60fc00-d091-462b-b897-f1b1b30d5361-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:19:11.981770 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.981749    2580 scope.go:117] "RemoveContainer" containerID="18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877"
Jul 16 21:19:11.982027 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:19:11.982008    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877\": container with ID starting with 18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877 not found: ID does not exist" containerID="18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877"
Jul 16 21:19:11.982069 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.982036    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877"} err="failed to get container status \"18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877\": rpc error: code = NotFound desc = could not find container \"18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877\": container with ID starting with 18517b467329c64b21b7af31732c0c11b5c62b14c0c483e154e698a813f2c877 not found: ID does not exist"
Jul 16 21:19:11.994446 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.994419    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"]
Jul 16 21:19:11.998244 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:11.998220    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-09402f-546bfd458d-mpwjk"]
Jul 16 21:19:13.502704 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:13.501020    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc60fc00-d091-462b-b897-f1b1b30d5361" path="/var/lib/kubelet/pods/dc60fc00-d091-462b-b897-f1b1b30d5361/volumes"
Jul 16 21:19:34.300544 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.298040    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"]
Jul 16 21:19:34.300544 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.298408    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428" podUID="dbd8cdb6-e206-4e82-822d-09a73b958361" containerName="maas-api" containerID="cri-o://bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37" gracePeriod=30
Jul 16 21:19:34.548974 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.548946    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:19:34.680411 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.680372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd8cdb6-e206-4e82-822d-09a73b958361-openshift-service-ca\") pod \"dbd8cdb6-e206-4e82-822d-09a73b958361\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") "
Jul 16 21:19:34.680411 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.680416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls\") pod \"dbd8cdb6-e206-4e82-822d-09a73b958361\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") "
Jul 16 21:19:34.680716 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.680446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqxm5\" (UniqueName: \"kubernetes.io/projected/dbd8cdb6-e206-4e82-822d-09a73b958361-kube-api-access-lqxm5\") pod \"dbd8cdb6-e206-4e82-822d-09a73b958361\" (UID: \"dbd8cdb6-e206-4e82-822d-09a73b958361\") "
Jul 16 21:19:34.680850 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.680826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd8cdb6-e206-4e82-822d-09a73b958361-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbd8cdb6-e206-4e82-822d-09a73b958361" (UID: "dbd8cdb6-e206-4e82-822d-09a73b958361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:19:34.682545 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.682497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbd8cdb6-e206-4e82-822d-09a73b958361" (UID: "dbd8cdb6-e206-4e82-822d-09a73b958361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:19:34.682651 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.682566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd8cdb6-e206-4e82-822d-09a73b958361-kube-api-access-lqxm5" (OuterVolumeSpecName: "kube-api-access-lqxm5") pod "dbd8cdb6-e206-4e82-822d-09a73b958361" (UID: "dbd8cdb6-e206-4e82-822d-09a73b958361"). InnerVolumeSpecName "kube-api-access-lqxm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:19:34.781619 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.781574    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd8cdb6-e206-4e82-822d-09a73b958361-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:19:34.781619 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.781611    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd8cdb6-e206-4e82-822d-09a73b958361-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:19:34.781619 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:34.781621    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqxm5\" (UniqueName: \"kubernetes.io/projected/dbd8cdb6-e206-4e82-822d-09a73b958361-kube-api-access-lqxm5\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 16 21:19:35.078995 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.078904    2580 generic.go:356] "Generic (PLEG): container finished" podID="dbd8cdb6-e206-4e82-822d-09a73b958361" containerID="bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37" exitCode=0
Jul 16 21:19:35.078995 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.078957    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"
Jul 16 21:19:35.079176 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.078959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428" event={"ID":"dbd8cdb6-e206-4e82-822d-09a73b958361","Type":"ContainerDied","Data":"bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37"}
Jul 16 21:19:35.079176 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.079074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428" event={"ID":"dbd8cdb6-e206-4e82-822d-09a73b958361","Type":"ContainerDied","Data":"098a5fc8742739a09c57d94454a9554ad95644309789aca7ecc54a30759d5468"}
Jul 16 21:19:35.079176 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.079094    2580 scope.go:117] "RemoveContainer" containerID="bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37"
Jul 16 21:19:35.088454 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.088428    2580 scope.go:117] "RemoveContainer" containerID="bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37"
Jul 16 21:19:35.088730 ip-10-0-140-200 kubenswrapper[2580]: E0716 21:19:35.088712    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37\": container with ID starting with bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37 not found: ID does not exist" containerID="bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37"
Jul 16 21:19:35.088815 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.088738    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37"} err="failed to get container status \"bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37\": rpc error: code = NotFound desc = could not find container \"bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37\": container with ID starting with bd7500e7c30f31baf3ee36dc97b9da73822a3aa6339e6be749ebabe7538e4a37 not found: ID does not exist"
Jul 16 21:19:35.104293 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.104248    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"]
Jul 16 21:19:35.109804 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.109779    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-69d8e7-66d99547d9-44428"]
Jul 16 21:19:35.498241 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:35.498208    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd8cdb6-e206-4e82-822d-09a73b958361" path="/var/lib/kubelet/pods/dbd8cdb6-e206-4e82-822d-09a73b958361/volumes"
Jul 16 21:19:52.572721 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:52.572685    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l4972_1d924c48-7c85-44fe-b87d-dea94fa4a233/manager/0.log"
Jul 16 21:19:52.689526 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:52.689477    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c97f6d9cb-c9ngr_ff7dd01c-c58b-4744-a59b-7f26b578e0ae/manager/0.log"
Jul 16 21:19:53.046552 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:53.046508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6477549b-4t5tr_77eeac63-d6a2-4459-9ac8-8af8351ddd02/copy-manifests/0.log"
Jul 16 21:19:53.053355 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:53.053323    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6477549b-4t5tr_77eeac63-d6a2-4459-9ac8-8af8351ddd02/manager/0.log"
Jul 16 21:19:53.704722 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:53.704687    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5898d5f5bf-hjllc_7e91e758-b977-4c63-a6be-6ab18bdf9f51/maas-api/0.log"
Jul 16 21:19:55.550569 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:55.550498    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wwb25_6e6d441d-0fb6-4548-beab-a8fb1ab6a0ca/postgres/0.log"
Jul 16 21:19:57.218290 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:57.218259    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qscrv_4e423712-9fa3-4d1b-8c2b-32ab61aba48b/kuadrant-console-plugin/0.log"
Jul 16 21:19:58.316585 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:58.316550    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-559f5f697c-jlhr5_139ab020-1707-4640-91b8-43b3e04d5e0f/kube-auth-proxy/0.log"
Jul 16 21:19:58.676944 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:58.676914    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-c9mwz_0da53d26-42a4-4390-ac54-12a019753193/payload-pre-processing/0.log"
Jul 16 21:19:59.253323 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:59.253294    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-d47ff_8a4472e6-ee63-4b40-a6e6-ae8cd66f6cae/main/0.log"
Jul 16 21:19:59.595853 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:19:59.595777    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz7f4b_8deb224b-0bc7-4761-aeab-5ef5b28ca136/main/0.log"
Jul 16 21:20:08.202503 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:08.202469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dv7n5_783a85c6-76a0-49bf-99fc-b1083bdc1bf7/global-pull-secret-syncer/0.log"
Jul 16 21:20:08.382548 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:08.382495    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qxzj8_e25e247f-b941-4c58-9bf0-61fb3ca27019/konnectivity-agent/0.log"
Jul 16 21:20:08.512157 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:08.512082    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-200.ec2.internal_a0b30ae5bc27eeb039f1a3250327b5a6/haproxy/0.log"
Jul 16 21:20:14.269350 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:14.269323    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qscrv_4e423712-9fa3-4d1b-8c2b-32ab61aba48b/kuadrant-console-plugin/0.log"
Jul 16 21:20:16.574408 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:16.574375    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-qbp74_04e72e36-5a2a-4975-9075-3c443ac41680/kube-state-metrics/0.log"
Jul 16 21:20:16.597789 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:16.597754    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-qbp74_04e72e36-5a2a-4975-9075-3c443ac41680/kube-rbac-proxy-main/0.log"
Jul 16 21:20:16.620004 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:16.619975    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-qbp74_04e72e36-5a2a-4975-9075-3c443ac41680/kube-rbac-proxy-self/0.log"
Jul 16 21:20:16.674576 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:16.674544    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-4l9jv_1b73f3dc-7f24-4aa8-b94b-bc0498ec898d/monitoring-plugin/0.log"
Jul 16 21:20:16.856348 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:16.856270    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdrch_16b48778-83ea-4905-bc1f-141bd634d46f/node-exporter/0.log"
Jul 16 21:20:16.879655 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:16.879632    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdrch_16b48778-83ea-4905-bc1f-141bd634d46f/kube-rbac-proxy/0.log"
Jul 16 21:20:16.903566 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:16.903537    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdrch_16b48778-83ea-4905-bc1f-141bd634d46f/init-textfile/0.log"
Jul 16 21:20:17.018305 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.018274    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e8da451-de2f-4eeb-8d17-176047e38b4d/prometheus/0.log"
Jul 16 21:20:17.046945 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.046915    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e8da451-de2f-4eeb-8d17-176047e38b4d/config-reloader/0.log"
Jul 16 21:20:17.073951 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.073924    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e8da451-de2f-4eeb-8d17-176047e38b4d/thanos-sidecar/0.log"
Jul 16 21:20:17.102426 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.102396    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e8da451-de2f-4eeb-8d17-176047e38b4d/kube-rbac-proxy-web/0.log"
Jul 16 21:20:17.139414 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.139387    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e8da451-de2f-4eeb-8d17-176047e38b4d/kube-rbac-proxy/0.log"
Jul 16 21:20:17.163687 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.163653    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e8da451-de2f-4eeb-8d17-176047e38b4d/kube-rbac-proxy-thanos/0.log"
Jul 16 21:20:17.192633 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.192600    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e8da451-de2f-4eeb-8d17-176047e38b4d/init-config-reloader/0.log"
Jul 16 21:20:17.221061 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.221024    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6pm2r_2038113a-1983-4fb3-93f9-ef7d9b40cd07/prometheus-operator/0.log"
Jul 16 21:20:17.242430 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.242397    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6pm2r_2038113a-1983-4fb3-93f9-ef7d9b40cd07/kube-rbac-proxy/0.log"
Jul 16 21:20:17.269980 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:17.269953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-tgrlf_5736e36f-2309-45b3-8051-9796822e2c5d/prometheus-operator-admission-webhook/0.log"
Jul 16 21:20:18.313073 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313037    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"]
Jul 16 21:20:18.313455 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313400    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd8cdb6-e206-4e82-822d-09a73b958361" containerName="maas-api"
Jul 16 21:20:18.313455 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313428    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd8cdb6-e206-4e82-822d-09a73b958361" containerName="maas-api"
Jul 16 21:20:18.313455 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313445    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba96290-bab9-4984-9e15-729ab49b047f" containerName="test-curl-7154-140581941127040"
Jul 16 21:20:18.313455 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313452    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba96290-bab9-4984-9e15-729ab49b047f" containerName="test-curl-7154-140581941127040"
Jul 16 21:20:18.313636 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313465    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc60fc00-d091-462b-b897-f1b1b30d5361" containerName="maas-api"
Jul 16 21:20:18.313636 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313471    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc60fc00-d091-462b-b897-f1b1b30d5361" containerName="maas-api"
Jul 16 21:20:18.313636 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313544    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba96290-bab9-4984-9e15-729ab49b047f" containerName="test-curl-7154-140581941127040"
Jul 16 21:20:18.313636 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313555    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc60fc00-d091-462b-b897-f1b1b30d5361" containerName="maas-api"
Jul 16 21:20:18.313636 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.313560    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd8cdb6-e206-4e82-822d-09a73b958361" containerName="maas-api"
Jul 16 21:20:18.317736 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.317717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.320762 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.320743    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rmclr\"/\"kube-root-ca.crt\""
Jul 16 21:20:18.321591 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.321577    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rmclr\"/\"default-dockercfg-g7srr\""
Jul 16 21:20:18.321659 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.321606    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rmclr\"/\"openshift-service-ca.crt\""
Jul 16 21:20:18.333608 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.333581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"]
Jul 16 21:20:18.353070 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.353038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-lib-modules\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.353201 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.353073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-podres\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.353201 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.353129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-sys\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.353201 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.353155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b2mg\" (UniqueName: \"kubernetes.io/projected/bd474e44-9bc2-4268-b144-75854d100ce1-kube-api-access-7b2mg\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.353308 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.353208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-proc\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454005 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.453968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-sys\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454160 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.454013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2mg\" (UniqueName: \"kubernetes.io/projected/bd474e44-9bc2-4268-b144-75854d100ce1-kube-api-access-7b2mg\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454160 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.454035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-proc\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454160 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.454098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-sys\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454160 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.454128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-proc\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454401 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.454210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-lib-modules\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454401 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.454254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-podres\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454401 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.454346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-lib-modules\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.454497 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.454405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd474e44-9bc2-4268-b144-75854d100ce1-podres\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.462601 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.462570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2mg\" (UniqueName: \"kubernetes.io/projected/bd474e44-9bc2-4268-b144-75854d100ce1-kube-api-access-7b2mg\") pod \"perf-node-gather-daemonset-7g9fb\" (UID: \"bd474e44-9bc2-4268-b144-75854d100ce1\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.627112 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.627019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:18.756368 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:18.756334    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"]
Jul 16 21:20:18.759135 ip-10-0-140-200 kubenswrapper[2580]: W0716 21:20:18.759107    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd474e44_9bc2_4268_b144_75854d100ce1.slice/crio-de24090e3afbfa05ca3c3aebabf8da9faae247dfe4f592fb719d2265d308246a WatchSource:0}: Error finding container de24090e3afbfa05ca3c3aebabf8da9faae247dfe4f592fb719d2265d308246a: Status 404 returned error can't find the container with id de24090e3afbfa05ca3c3aebabf8da9faae247dfe4f592fb719d2265d308246a
Jul 16 21:20:19.285025 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:19.284989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb" event={"ID":"bd474e44-9bc2-4268-b144-75854d100ce1","Type":"ContainerStarted","Data":"135e41fa3b687ee240ed9e76b72cff7abc614b6ae3dc04969ec0254a8c453742"}
Jul 16 21:20:19.285203 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:19.285041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb" event={"ID":"bd474e44-9bc2-4268-b144-75854d100ce1","Type":"ContainerStarted","Data":"de24090e3afbfa05ca3c3aebabf8da9faae247dfe4f592fb719d2265d308246a"}
Jul 16 21:20:19.285203 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:19.285071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:19.301279 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:19.301233    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb" podStartSLOduration=1.301218253 podStartE2EDuration="1.301218253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:20:19.300092159 +0000 UTC m=+2826.403371145" watchObservedRunningTime="2026-07-16 21:20:19.301218253 +0000 UTC m=+2826.404497238"
Jul 16 21:20:19.666610 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:19.666578    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-h6r9r_30ab91e5-66ba-426d-bd20-e4767ef0202c/download-server/0.log"
Jul 16 21:20:21.270402 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:21.270373    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lvz59_406a92af-2a2b-40c8-8026-a456c8de1b42/dns/0.log"
Jul 16 21:20:21.292798 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:21.292769    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lvz59_406a92af-2a2b-40c8-8026-a456c8de1b42/kube-rbac-proxy/0.log"
Jul 16 21:20:21.421770 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:21.421740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x2tl9_93410d88-85b1-4b03-b66b-d6de95740d86/dns-node-resolver/0.log"
Jul 16 21:20:22.013732 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:22.013701    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rnthw_24a3af80-d268-4bb1-8a80-b6a69d5d6db2/node-ca/0.log"
Jul 16 21:20:23.155328 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:23.155300    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-559f5f697c-jlhr5_139ab020-1707-4640-91b8-43b3e04d5e0f/kube-auth-proxy/0.log"
Jul 16 21:20:23.275845 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:23.275816    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-c9mwz_0da53d26-42a4-4390-ac54-12a019753193/payload-pre-processing/0.log"
Jul 16 21:20:24.056831 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:24.056797    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mw784_3b97356d-6fe0-4246-bdf7-f533a896daea/serve-healthcheck-canary/0.log"
Jul 16 21:20:24.567755 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:24.567727    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5htcq_dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3/kube-rbac-proxy/0.log"
Jul 16 21:20:24.588321 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:24.588294    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5htcq_dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3/exporter/0.log"
Jul 16 21:20:24.610794 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:24.610761    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5htcq_dd44ffcb-1f3b-4b0d-a1ac-e7d26171e8a3/extractor/0.log"
Jul 16 21:20:25.299575 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:25.299549    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-7g9fb"
Jul 16 21:20:27.195312 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:27.195198    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l4972_1d924c48-7c85-44fe-b87d-dea94fa4a233/manager/0.log"
Jul 16 21:20:27.306293 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:27.306265    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c97f6d9cb-c9ngr_ff7dd01c-c58b-4744-a59b-7f26b578e0ae/manager/0.log"
Jul 16 21:20:27.434806 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:27.434780    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6477549b-4t5tr_77eeac63-d6a2-4459-9ac8-8af8351ddd02/manager/0.log"
Jul 16 21:20:27.457091 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:27.457065    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6477549b-4t5tr_77eeac63-d6a2-4459-9ac8-8af8351ddd02/copy-manifests/0.log"
Jul 16 21:20:28.925209 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:28.925182    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-559f5f697c-jlhr5_139ab020-1707-4640-91b8-43b3e04d5e0f/kube-auth-proxy/0.log"
Jul 16 21:20:29.044565 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:29.044538    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-649bd55f77-ffth9_28cbdb0a-9644-497e-aa48-78f2f020340f/manager/0.log"
Jul 16 21:20:29.068131 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:29.068104    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-c9mwz_0da53d26-42a4-4390-ac54-12a019753193/payload-pre-processing/0.log"
Jul 16 21:20:30.575386 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:30.575359    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-559f5f697c-jlhr5_139ab020-1707-4640-91b8-43b3e04d5e0f/kube-auth-proxy/0.log"
Jul 16 21:20:30.673611 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:30.673579    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-c9mwz_0da53d26-42a4-4390-ac54-12a019753193/payload-pre-processing/0.log"
Jul 16 21:20:34.324073 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:34.324043    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-7p49b_bea279e7-635d-483f-9809-301988f923a7/migrator/0.log"
Jul 16 21:20:34.344094 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:34.344063    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-7p49b_bea279e7-635d-483f-9809-301988f923a7/graceful-termination/0.log"
Jul 16 21:20:35.956664 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:35.956634    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sr7vx_230c5d01-b3a9-4a57-a642-4ac6905b4b4b/kube-multus-additional-cni-plugins/0.log"
Jul 16 21:20:35.978741 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:35.978712    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sr7vx_230c5d01-b3a9-4a57-a642-4ac6905b4b4b/egress-router-binary-copy/0.log"
Jul 16 21:20:36.000217 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:36.000181    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sr7vx_230c5d01-b3a9-4a57-a642-4ac6905b4b4b/cni-plugins/0.log"
Jul 16 21:20:36.021407 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:36.021379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sr7vx_230c5d01-b3a9-4a57-a642-4ac6905b4b4b/bond-cni-plugin/0.log"
Jul 16 21:20:36.043096 ip-10-0-140-200 kubenswrapper[2580]: I0716 21:20:36.043067    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sr7vx_230c5d01-b3a9-4a57-a642-4ac6905b4b4b/routeoverride-cni/0.log"