Sep 08 18:23:37.483720 ip-10-0-141-6 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 18:23:37.910597 ip-10-0-141-6 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:23:37.910597 ip-10-0-141-6 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 18:23:37.910597 ip-10-0-141-6 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:23:37.910597 ip-10-0-141-6 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 18:23:37.910597 ip-10-0-141-6 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:23:37.912526 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.912437    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 18:23:37.917648 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917620    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:37.917648 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917641    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:37.917648 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917647    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:37.917648 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917651    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:37.917648 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917655    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917659    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917663    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917666    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917670    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917675    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917679    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917682    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917686    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917689    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917692    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917696    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917700    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917703    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917707    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917711    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917714    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917719    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917722    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:37.918010 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917726    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917730    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917734    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917739    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917744    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917748    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917752    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917756    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917759    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917763    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917767    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917771    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917778    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917784    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917788    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917794    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917799    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917803    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917807    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917812    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:37.918602 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917816    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917820    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917823    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917828    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917831    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917836    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917841    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917847    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917851    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917856    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917860    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917864    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917868    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917872    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917876    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917880    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917884    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917888    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917892    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917896    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:37.919115 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917901    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917905    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917909    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917912    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917917    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917921    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917925    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917930    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917934    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917938    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917942    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917951    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917957    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917962    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917966    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917970    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917974    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917978    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917982    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917986    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:37.919961 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917991    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.917997    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918003    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918006    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918011    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918016    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918020    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918025    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918029    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918033    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918037    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918041    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918045    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918049    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918054    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.918058    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920219    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920230    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920236    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:37.920855 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920241    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920245    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920250    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920254    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920259    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920263    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920268    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920272    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920276    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920280    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920284    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920288    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920292    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920296    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920301    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920308    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920312    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920316    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920320    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920325    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:37.921698 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920329    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920334    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920338    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920342    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920346    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920350    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920354    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920359    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920363    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920367    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920371    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920376    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920380    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920384    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920388    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920392    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920397    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920402    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920405    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920410    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:37.922670 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920415    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920419    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920445    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920450    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920454    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920458    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920462    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920468    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920472    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920476    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920480    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920484    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920487    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920492    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920496    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920500    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920504    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920508    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920513    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920517    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:37.923604 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920521    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920525    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920529    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920534    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920538    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920542    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920546    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920551    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920555    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920559    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920563    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920567    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920572    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920576    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920580    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920584    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920591    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920596    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920601    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:37.924116 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920606    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920614    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920621    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920625    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920632    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920636    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920640    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920644    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920648    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920652    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920656    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920660    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920666    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920672    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920676    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920680    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.920684    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921329    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921340    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:37.924700 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921346    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921351    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921355    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921360    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921365    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921368    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921373    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921377    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921381    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921385    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921389    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921394    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921398    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921402    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921407    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921411    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921416    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921420    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921442    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921447    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:37.925216 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921452    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921456    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921460    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921464    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921468    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921472    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921476    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921481    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921485    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921489    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921493    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921497    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921501    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921505    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921510    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921514    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921518    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921522    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921527    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921531    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:37.925757 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921535    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921539    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921543    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921547    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921551    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921555    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921559    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921564    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921568    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921572    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921576    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921580    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921584    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921589    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921593    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921599    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921605    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921611    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921616    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:37.926485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921621    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921625    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921630    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921634    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921639    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921643    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921647    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921651    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921655    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921659    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921663    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921670    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921674    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921678    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921684    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921690    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921694    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921698    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921703    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:37.926960 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921707    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921711    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921715    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921719    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921734    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921740    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921744    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921748    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921753    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921757    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921761    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921765    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921769    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921772    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921776    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921780    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921785    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921789    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.921793    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921895    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 18:23:37.927447 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921952    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921961    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921967    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921974    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921981    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921987    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921994    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.921999    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922004    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922010    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922015    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922020    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922025    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922030    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922034    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922040    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922045    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922052    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922057    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922062    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922067    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922072    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922078    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 18:23:37.928047 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922083    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922088    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922093    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922097    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922102    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922106    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922112    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922117    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922125    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922129    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922134    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922139    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922144    2580 flags.go:64] FLAG: --enable-server="true"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922150    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922157    2580 flags.go:64] FLAG: --event-burst="100"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922162    2580 flags.go:64] FLAG: --event-qps="50"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922167    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922172    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922176    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922182    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922187    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922192    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922196    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922201    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922206    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 18:23:37.928705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922211    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922216    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922221    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922225    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922230    2580 flags.go:64] FLAG: --feature-gates=""
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922236    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922240    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922246    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922251    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922255    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922260    2580 flags.go:64] FLAG: --help="false"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922264    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-141-6.ec2.internal"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922269    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922276    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922281    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922286    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922292    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922297    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922301    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922306    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922311    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922316    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922321    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922326    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 08 18:23:37.929352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922330    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922335    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922340    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922344    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922349    2580 flags.go:64] FLAG: --lock-file=""
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922354    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922358    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922363    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922371    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922382    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922387    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922391    2580 flags.go:64] FLAG: --logging-format="text"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922396    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922401    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922406    2580 flags.go:64] FLAG: --manifest-url=""
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922410    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922417    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922422    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922449    2580 flags.go:64] FLAG: --max-pods="110"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922453    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922457    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922463    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922468    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922473    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922478    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 18:23:37.929975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922483    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922494    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922498    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922504    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922527    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922533    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922542    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922547    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922552    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922556    2580 flags.go:64] FLAG: --port="10250"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922561    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922566    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d9cb7c8728912eae"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922571    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922576    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922580    2580 flags.go:64] FLAG: --register-node="true"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922585    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922591    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922597    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922601    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922606    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922611    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922617    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922621    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922626    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922630    2580 flags.go:64] FLAG: --runonce="false"
Sep 08 18:23:37.930639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922635    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922640    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922645    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922649    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922655    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922660    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922665    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922669    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922673    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922678    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922683    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922687    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922692    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922696    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922701    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922710    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922714    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922719    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922726    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922731    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922735    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922740    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922744    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922749    2580 flags.go:64] FLAG: --v="2"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922755    2580 flags.go:64] FLAG: --version="false"
Sep 08 18:23:37.931286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922763    2580 flags.go:64] FLAG: --vmodule=""
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922769    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.922774    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922955    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922962    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922969    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922977    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922981    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922985    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922989    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922993    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.922999    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923003    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923007    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923011    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923015    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923018    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923023    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923027    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923030    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:37.931933 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923034    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923038    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923042    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923046    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923049    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923053    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923057    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923061    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923065    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923071    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923074    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923078    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923082    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923086    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923090    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923094    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923098    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923102    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923106    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923110    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:37.932484 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923114    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923118    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923122    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923128    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923132    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923136    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923140    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923144    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923147    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923152    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923158    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923165    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923170    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923175    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923180    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923184    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923188    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923192    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923197    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:37.932981 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923201    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923208    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923214    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923219    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923223    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923227    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923232    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923238    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923243    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923247    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923252    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923256    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923260    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923265    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923268    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923272    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923279    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923283    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923287    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923291    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:37.933550 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923296    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923300    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923304    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923308    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923312    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923316    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923320    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923324    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923328    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923332    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923336    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923340    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923344    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923348    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923352    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923356    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923360    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923365    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923370    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:37.934036 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923374    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923379    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923382    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.923386    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.924142    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.932232    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.932249    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.932270    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.932279    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932334    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932339    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932343    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932346    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932349    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:37.934536 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932352    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932354    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932357    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932360    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932362    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932365    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932367    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932370    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932372    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932374    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932377    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932380    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932382    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932385    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932387    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932389    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932392    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932394    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932397    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932399    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932402    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:37.934901 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932404    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932408    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932412    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932416    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932419    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932422    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932456    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932459    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932463    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932467    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932472    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932475    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932478    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932481    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932483    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932486    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932489    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932492    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932494    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:37.935412 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932497    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932500    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932503    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932505    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932508    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932511    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932513    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932516    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932518    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932521    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932523    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932526    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932528    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932531    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932533    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932536    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932540    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932544    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932547    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932550    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:37.935993 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932552    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932555    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932558    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932560    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932563    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932566    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932568    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932570    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932573    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932575    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932578    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932581    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932583    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932585    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932588    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932590    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932592    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932595    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932597    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:37.936525 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932600    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932602    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932604    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932607    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932610    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932612    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932615    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932617    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932620    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932623    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932626    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932629    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932632    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932634    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:37.937096 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932637    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.932642    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932753    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932757    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932760    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932763    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932766    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932768    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932771    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932775    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932778    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932781    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932784    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932787    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932790    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:37.937485 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932792    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932794    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932797    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932799    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932802    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932804    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932807    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932809    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932812    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932814    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932817    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932819    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932822    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932824    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932827    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932830    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932832    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932834    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932837    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932840    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:37.937953 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932842    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932845    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932847    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932849    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932852    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932854    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932856    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932859    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932861    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932863    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932866    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932868    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932871    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932873    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932876    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932878    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932880    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932883    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932885    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:37.938468 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932888    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932890    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932893    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932895    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932898    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932900    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932903    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932905    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932908    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932911    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932913    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932916    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932918    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932921    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932924    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932927    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932929    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932932    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932936    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:37.939058 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932939    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932942    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932944    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932947    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932950    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932953    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932955    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932958    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932961    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932964    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932967    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932970    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932973    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932975    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932978    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932980    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932983    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932985    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932988    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932991    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:37.939860 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932993    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932996    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.932998    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.933002    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.933004    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.933007    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.933009    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:37.933012    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.933017    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.933123    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.935620    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.936685    2580 server.go:1019] "Starting client certificate rotation"
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.936793    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 18:23:37.940475 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.936838    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 18:23:37.941006 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.939515    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 18:23:37.946665 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.946638    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 18:23:37.956354 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.956334    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 08 18:23:37.959944 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.959927    2580 log.go:25] "Validated CRI v1 image API"
Sep 08 18:23:37.960874 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.960857    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 18:23:37.968071 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.968047    2580 fs.go:135] Filesystem UUIDs: map[4b645523-8a78-4e94-a32e-87f73259c7cd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bad33c3b-614b-4a00-803a-852f4539ed8d:/dev/nvme0n1p3]
Sep 08 18:23:37.968156 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.968069    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 18:23:37.971307 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.971285    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 18:23:37.973254 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.973116    2580 manager.go:216] Machine: {Timestamp:2026-09-08 18:23:37.97171174 +0000 UTC m=+0.382021813 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099581 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2660e93a27922fcdb0528745f25750 SystemUUID:ec2660e9-3a27-922f-cdb0-528745f25750 BootID:33d24193-59ca-447b-9a8f-818b147cb581 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:c9:4d:a4:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:c9:4d:a4:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:9d:f4:c2:08:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 18:23:37.973254 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.973245    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 18:23:37.973372 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.973333    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 18:23:37.976848 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.976822    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 18:23:37.977022 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.976851    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 18:23:37.978462 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.978450    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 18:23:37.978521 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.978470    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 18:23:37.978521 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.978482    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 18:23:37.979343 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.979331    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 18:23:37.981323 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.981311    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 18:23:37.986012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.985998    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 18:23:37.986143 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.986132    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 18:23:37.986199 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.986173    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 18:23:37.986199 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.986188    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 18:23:37.986291 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.986207    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 18:23:37.986291 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.986221    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 08 18:23:37.986291 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.986233    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 18:23:37.986623 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.986610    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 18:23:37.986685 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.986632    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 18:23:37.989753 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.989736    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 18:23:37.992660 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.992646    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 18:23:37.992726 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.992676    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 18:23:37.992866 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.992854    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 18:23:37.992907 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.992871    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 18:23:37.992907 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.992879    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 18:23:37.992907 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.992884    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 18:23:37.992907 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.992891    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 18:23:37.992907 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.992896    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 18:23:37.993512 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.993501    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 18:23:37.993512 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.993512    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 18:23:37.993571 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.993519    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 18:23:37.993571 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.993526    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 18:23:37.993571 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.993534    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 18:23:37.993571 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.993542    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 18:23:37.994481 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.994470    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 18:23:37.994519 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.994482    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 18:23:37.996513 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:37.996480    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 18:23:37.996601 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:37.996490    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 18:23:37.996842 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.996819    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 18:23:37.999350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.999333    2580 server.go:1268] "Started kubelet"
Sep 08 18:23:37.999519 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.999449    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 18:23:37.999613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.999448    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 18:23:37.999613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:37.999582    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 18:23:38.000285 ip-10-0-141-6 systemd[1]: Started Kubernetes Kubelet.
Sep 08 18:23:38.001053 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.001030    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 18:23:38.001893 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.001878    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 18:23:38.005258 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.005240    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 18:23:38.006957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.006937    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 18:23:38.007689 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.007675    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 18:23:38.007781 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.007748    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.008466 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.008447    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 18:23:38.008466 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.008449    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 18:23:38.008607 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.008475    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 18:23:38.008607 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.008568    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 18:23:38.008607 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.008578    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 18:23:38.008607 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.008593    2580 factory.go:55] Registering systemd factory
Sep 08 18:23:38.008607 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.008609    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 08 18:23:38.009074 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.008952    2580 factory.go:153] Registering CRI-O factory
Sep 08 18:23:38.009132 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.009078    2580 factory.go:223] Registration of the crio container factory successfully
Sep 08 18:23:38.009132 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.009130    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 18:23:38.009235 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.009152    2580 factory.go:103] Registering Raw factory
Sep 08 18:23:38.009235 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.009176    2580 manager.go:1196] Started watching for new ooms in manager
Sep 08 18:23:38.009522 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.009510    2580 manager.go:349] Starting recovery of all containers
Sep 08 18:23:38.016985 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.016958    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 18:23:38.017072 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.017021    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 18:23:38.018109 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.016845    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-6.ec2.internal.18d36ba169735320  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-6.ec2.internal,UID:ip-10-0-141-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-6.ec2.internal,},FirstTimestamp:2026-09-08 18:23:37.999291168 +0000 UTC m=+0.409601241,LastTimestamp:2026-09-08 18:23:37.999291168 +0000 UTC m=+0.409601241,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-6.ec2.internal,}"
Sep 08 18:23:38.020784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.020764    2580 manager.go:354] Recovery completed
Sep 08 18:23:38.025864 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.025851    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:38.026060 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.026043    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxzcg"
Sep 08 18:23:38.028059 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.028041    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:38.028124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.028070    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:38.028124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.028082    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:38.028652 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.028640    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 18:23:38.028652 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.028650    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 18:23:38.028744 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.028658    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:38.028744 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.028664    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 18:23:38.029511 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.029495    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:38.029576 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.029521    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:38.029576 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.029532    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:38.029657 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.029593    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-6.ec2.internal.18d36ba16b2a415f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-6.ec2.internal,UID:ip-10-0-141-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-6.ec2.internal,},FirstTimestamp:2026-09-08 18:23:38.028056927 +0000 UTC m=+0.438366997,LastTimestamp:2026-09-08 18:23:38.028056927 +0000 UTC m=+0.438366997,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-6.ec2.internal,}"
Sep 08 18:23:38.030854 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.030839    2580 policy_none.go:49] "None policy: Start"
Sep 08 18:23:38.030918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.030858    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 18:23:38.030918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.030871    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 18:23:38.031798 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.031782    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxzcg"
Sep 08 18:23:38.032966 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.032949    2580 policy_none.go:47] "Start"
Sep 08 18:23:38.073001 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.072982    2580 manager.go:337] "Starting Device Plugin manager"
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.073028    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.073042    2580 server.go:87] "Starting device plugin registration server"
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.073399    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.073411    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.073519    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.073608    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.073617    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.074230    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 18:23:38.088317 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.074291    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.112570 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.112536    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 18:23:38.113868 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.113853    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 18:23:38.113961 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.113910    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 18:23:38.113961 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.113944    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 18:23:38.114059 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.113991    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 18:23:38.116242 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.116218    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:38.174407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.174276    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:38.175286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.175267    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:38.175375 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.175298    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:38.175375 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.175309    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:38.175375 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.175331    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.189831 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.189812    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.189933 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.189835    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-6.ec2.internal\": node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.213955 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.213934    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.214946 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.214927    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"]
Sep 08 18:23:38.215003 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.214988    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:38.216503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.216487    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:38.216578 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.216515    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:38.216578 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.216524    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:38.218795 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.218780    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:38.218912 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.218892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.218949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.218939    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:38.219570 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.219547    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:38.219570 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.219559    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:38.219699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.219575    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:38.219699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.219582    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:38.219699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.219584    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:38.219699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.219592    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:38.221859 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.221841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.221945 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.221866    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:38.222581 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.222566    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:38.222667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.222597    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:38.222667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.222611    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:38.251256 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.251234    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-6.ec2.internal\" not found" node="ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.255780 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.255764    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-6.ec2.internal\" not found" node="ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.310906 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.310879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19d2e81f86efd718848a5fc68affb80f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-6.ec2.internal\" (UID: \"19d2e81f86efd718848a5fc68affb80f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.311008 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.310909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bff7db94e64466d52b7d7679d6fa34c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"1bff7db94e64466d52b7d7679d6fa34c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.311008 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.310926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bff7db94e64466d52b7d7679d6fa34c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"1bff7db94e64466d52b7d7679d6fa34c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.314564 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.314548    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.411519 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.411490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19d2e81f86efd718848a5fc68affb80f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-6.ec2.internal\" (UID: \"19d2e81f86efd718848a5fc68affb80f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.411627 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.411450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19d2e81f86efd718848a5fc68affb80f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-6.ec2.internal\" (UID: \"19d2e81f86efd718848a5fc68affb80f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.411627 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.411565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bff7db94e64466d52b7d7679d6fa34c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"1bff7db94e64466d52b7d7679d6fa34c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.411627 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.411545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bff7db94e64466d52b7d7679d6fa34c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"1bff7db94e64466d52b7d7679d6fa34c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.411627 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.411591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bff7db94e64466d52b7d7679d6fa34c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"1bff7db94e64466d52b7d7679d6fa34c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.411627 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.411610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bff7db94e64466d52b7d7679d6fa34c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"1bff7db94e64466d52b7d7679d6fa34c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.415559 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.415539    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.516369 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.516315    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.554553 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.554526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.558259 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.558240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:38.616831 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.616794    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.717297 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.717267    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.817785 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.817717    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.918327 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:38.918298    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:38.936463 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.936446    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 18:23:38.936604 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:38.936587    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 18:23:39.014809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.014785    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:39.018409 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:39.018391    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:39.034582 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.034521    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 18:18:39 +0000 UTC" deadline="2028-03-16 21:46:45.635925158 +0000 UTC"
Sep 08 18:23:39.034582 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.034579    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13323h23m6.60134894s"
Sep 08 18:23:39.119007 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:39.118949    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:39.137652 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:39.137619    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bff7db94e64466d52b7d7679d6fa34c.slice/crio-c9fca7a9133f9b52edca45cfe76e7d8060cc12a27eaad7fcea50f4b6b5724da6 WatchSource:0}: Error finding container c9fca7a9133f9b52edca45cfe76e7d8060cc12a27eaad7fcea50f4b6b5724da6: Status 404 returned error can't find the container with id c9fca7a9133f9b52edca45cfe76e7d8060cc12a27eaad7fcea50f4b6b5724da6
Sep 08 18:23:39.145181 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.145158    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:23:39.150247 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:39.150229    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d2e81f86efd718848a5fc68affb80f.slice/crio-50a9603211f3ce97234bace53449ae6b700ae6b5e71024adad79d004be7b5779 WatchSource:0}: Error finding container 50a9603211f3ce97234bace53449ae6b700ae6b5e71024adad79d004be7b5779: Status 404 returned error can't find the container with id 50a9603211f3ce97234bace53449ae6b700ae6b5e71024adad79d004be7b5779
Sep 08 18:23:39.219703 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:39.219671    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:39.315888 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.315867    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:39.320222 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:39.320208    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:39.420830 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:39.420764    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found"
Sep 08 18:23:39.482650 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.482628    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:39.508976 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.508955    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:39.519010 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.518988    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 18:23:39.520031 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.520020    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal"
Sep 08 18:23:39.531059 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.531038    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 18:23:39.987566 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:39.987489    2580 apiserver.go:52] "Watching apiserver"
Sep 08 18:23:40.003128 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.003094    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 18:23:40.003904 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.003536    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-m824h","kube-system/konnectivity-agent-xv525","openshift-cluster-node-tuning-operator/tuned-pmgnl","openshift-dns/node-resolver-lgmvk","openshift-image-registry/node-ca-gb8zd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal","openshift-multus/multus-additional-cni-plugins-cvhnn","openshift-ovn-kubernetes/ovnkube-node-wvdhn","kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k","openshift-multus/multus-29bnw","openshift-multus/network-metrics-daemon-ndrsb","openshift-network-diagnostics/network-check-target-tgnbz","openshift-network-operator/iptables-alerter-rv7d7"]
Sep 08 18:23:40.006318 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.006296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.007760 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.007695    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 18:23:40.008985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.008953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:40.009083 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.009042    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:40.010605 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.010583    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 18:23:40.010698 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.010662    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 18:23:40.012260 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.012238    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 18:23:40.012532 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.012496    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7fqs4\""
Sep 08 18:23:40.012744 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.012729    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 18:23:40.012978 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.012960    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 18:23:40.014934 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.014554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.017299 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.017281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.018625    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.018682    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.018629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qsl94\""
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-cni-binary-copy\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-sys\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad692771-2e44-445c-b28d-1e0b7ffedaac-tmp\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-cnibin\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-kubernetes\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysctl-d\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-lib-modules\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-var-lib-kubelet\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysconfig\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysctl-conf\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.020108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-systemd\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-host\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.019955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkljj\" (UniqueName: \"kubernetes.io/projected/ad692771-2e44-445c-b28d-1e0b7ffedaac-kube-api-access-tkljj\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.020014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-system-cni-dir\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.020040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdnv\" (UniqueName: \"kubernetes.io/projected/852cc4a9-da83-4816-a835-80bd44ada77f-kube-api-access-4zdnv\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.020082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-modprobe-d\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.020104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-run\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.020126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-tuned\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.020320    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.020544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.020640    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jx2bc\""
Sep 08 18:23:40.021169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.021051    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 18:23:40.021803 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.021504    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 18:23:40.022556 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.022533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.023129 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.023109    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 18:23:40.023621 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.023594    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 18:23:40.023968 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.023949    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqxjc\""
Sep 08 18:23:40.024261 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.024139    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 18:23:40.025183 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.025166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.026188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.025431    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:23:40.026188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.025784    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 18:23:40.026188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.025246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 18:23:40.026762 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.026585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7rvk\""
Sep 08 18:23:40.028480 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.027886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.028480 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.027953    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:40.028480 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.028259    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 18:23:40.029416 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.029390    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 18:23:40.029699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.029679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 18:23:40.029931 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.029914    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 18:23:40.030255 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.030229    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 18:23:40.030500 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.030480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 18:23:40.031355 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.031317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbh94\""
Sep 08 18:23:40.031719 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.031699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:23:40.033339 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.033311    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.035226 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.035201    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jhtvb\""
Sep 08 18:23:40.035319 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.035290    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 18:23:40.035707 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.035687    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 18:23:40.037051 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.037028    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 18:23:40.037383 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.037367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 18:23:40.037887 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.037868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c6gtv\""
Sep 08 18:23:40.039640 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.039524    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.042540 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.042495    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 18:23:40.042811 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.042797    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zsjpb\""
Sep 08 18:23:40.043484 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.043464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:40.043572 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.043553    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:40.044028 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.043856    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 18:23:40.055199 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.055177    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9mf79"
Sep 08 18:23:40.066852 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.066830    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9mf79"
Sep 08 18:23:40.110393 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.110371    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 18:23:40.120318 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.120292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad692771-2e44-445c-b28d-1e0b7ffedaac-tmp\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.120532 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.120510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-kubelet\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.120746 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.120701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" event={"ID":"19d2e81f86efd718848a5fc68affb80f","Type":"ContainerStarted","Data":"50a9603211f3ce97234bace53449ae6b700ae6b5e71024adad79d004be7b5779"}
Sep 08 18:23:40.120836 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.120802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-ovn\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.120897 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.120836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-cni-bin\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.120959 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.120898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-kubelet\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.121015 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.120972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-conf-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.121015 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.121001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-kubernetes\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.121117 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.121060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-var-lib-kubelet\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.121185 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.121159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-kubernetes\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.121260 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.121224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-var-lib-kubelet\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.121891 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.121868    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 18:23:40.123400 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.121090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.123400 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cef15672-783f-42d2-a325-88f90a658e96-agent-certs\") pod \"konnectivity-agent-xv525\" (UID: \"cef15672-783f-42d2-a325-88f90a658e96\") " pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:23:40.123400 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysconfig\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.123400 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysctl-conf\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-systemd\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-systemd-units\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8e9951e-e709-431d-9017-10e337c05160-host-slash\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-sys\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-cni-binary-copy\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-kubelet-config\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-dbus\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-socket-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.123674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysctl-conf\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-cnibin\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-systemd\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysctl-d\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkljj\" (UniqueName: \"kubernetes.io/projected/ad692771-2e44-445c-b28d-1e0b7ffedaac-kube-api-access-tkljj\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-slash\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-ovnkube-config\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysctl-d\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-cni-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmqk\" (UniqueName: \"kubernetes.io/projected/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-kube-api-access-nqmqk\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.123979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-host\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-systemd\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-ovnkube-script-lib\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-k8s-cni-cncf-io\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqpql\" (UniqueName: \"kubernetes.io/projected/42d754f9-b01a-4e28-822e-79349dec9556-kube-api-access-lqpql\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8e9951e-e709-431d-9017-10e337c05160-iptables-alerter-script\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.124210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-sys-fs\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdnv\" (UniqueName: \"kubernetes.io/projected/852cc4a9-da83-4816-a835-80bd44ada77f-kube-api-access-4zdnv\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-run\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-var-lib-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-multus-certs\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-cni-binary-copy\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-sys\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-sysconfig\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdh22\" (UniqueName: \"kubernetes.io/projected/31debb12-8b51-43d8-8c51-4bba3cd84bcd-kube-api-access-wdh22\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-lib-modules\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-run\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-run-netns\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-log-socket\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-cni-netd\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1106185-4212-4823-9a16-799768b162ff-ovn-node-metrics-cert\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjz9\" (UniqueName: \"kubernetes.io/projected/b1106185-4212-4823-9a16-799768b162ff-kube-api-access-2vjz9\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-netns\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.124986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-host\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" event={"ID":"1bff7db94e64466d52b7d7679d6fa34c","Type":"ContainerStarted","Data":"c9fca7a9133f9b52edca45cfe76e7d8060cc12a27eaad7fcea50f4b6b5724da6"}
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-cnibin\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-hostroot\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-etc-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cef15672-783f-42d2-a325-88f90a658e96-konnectivity-ca\") pod \"konnectivity-agent-xv525\" (UID: \"cef15672-783f-42d2-a325-88f90a658e96\") " pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-device-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-lib-modules\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-system-cni-dir\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-tuned\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-env-overrides\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-system-cni-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-socket-dir-parent\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.125955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-etc-kubernetes\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3337f71d-84e5-4047-b6e3-614c0143b182-serviceca\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbn8f\" (UniqueName: \"kubernetes.io/projected/a8e9951e-e709-431d-9017-10e337c05160-kube-api-access-xbn8f\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-cni-binary-copy\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-registration-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/852cc4a9-da83-4816-a835-80bd44ada77f-system-cni-dir\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31debb12-8b51-43d8-8c51-4bba3cd84bcd-tmp-dir\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-node-log\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3337f71d-84e5-4047-b6e3-614c0143b182-host\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-942gj\" (UniqueName: \"kubernetes.io/projected/3337f71d-84e5-4047-b6e3-614c0143b182-kube-api-access-942gj\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-cni-bin\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.125984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31debb12-8b51-43d8-8c51-4bba3cd84bcd-hosts-file\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.126739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-modprobe-d\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.127381 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-cnibin\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.127381 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-cni-multus\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.127381 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-daemon-config\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.127381 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.127381 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x47s\" (UniqueName: \"kubernetes.io/projected/798748c6-4900-47af-b91d-12f8468fb97d-kube-api-access-7x47s\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.127381 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-modprobe-d\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.127381 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.126666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/852cc4a9-da83-4816-a835-80bd44ada77f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.130357 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.130165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad692771-2e44-445c-b28d-1e0b7ffedaac-etc-tuned\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.130463 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.130204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad692771-2e44-445c-b28d-1e0b7ffedaac-tmp\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.130463 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.130260    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:40.130463 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.130430    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:40.130463 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.130446    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2t2p9 for pod openshift-network-diagnostics/network-check-target-tgnbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:40.130699 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.130525    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9 podName:24ca9f64-7590-47f3-8925-43b6f28056e2 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:40.630487967 +0000 UTC m=+3.040798045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2t2p9" (UniqueName: "kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9") pod "network-check-target-tgnbz" (UID: "24ca9f64-7590-47f3-8925-43b6f28056e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:40.133323 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.133292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkljj\" (UniqueName: \"kubernetes.io/projected/ad692771-2e44-445c-b28d-1e0b7ffedaac-kube-api-access-tkljj\") pod \"tuned-pmgnl\" (UID: \"ad692771-2e44-445c-b28d-1e0b7ffedaac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.133424 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.133327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdnv\" (UniqueName: \"kubernetes.io/projected/852cc4a9-da83-4816-a835-80bd44ada77f-kube-api-access-4zdnv\") pod \"multus-additional-cni-plugins-cvhnn\" (UID: \"852cc4a9-da83-4816-a835-80bd44ada77f\") " pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.227027 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.226989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-cni-binary-copy\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.227027 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-kubelet-config\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-dbus\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-socket-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-slash\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-ovnkube-config\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-cni-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmqk\" (UniqueName: \"kubernetes.io/projected/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-kube-api-access-nqmqk\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-systemd\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-ovnkube-script-lib\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-k8s-cni-cncf-io\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqpql\" (UniqueName: \"kubernetes.io/projected/42d754f9-b01a-4e28-822e-79349dec9556-kube-api-access-lqpql\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8e9951e-e709-431d-9017-10e337c05160-iptables-alerter-script\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-sys-fs\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.227292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-var-lib-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-multus-certs\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdh22\" (UniqueName: \"kubernetes.io/projected/31debb12-8b51-43d8-8c51-4bba3cd84bcd-kube-api-access-wdh22\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-run-netns\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-log-socket\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-cni-netd\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-slash\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-systemd\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1106185-4212-4823-9a16-799768b162ff-ovn-node-metrics-cert\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-k8s-cni-cncf-io\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.227889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-dbus\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-run-netns\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.227959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-log-socket\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.227973    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.228058    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs podName:42d754f9-b01a-4e28-822e-79349dec9556 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:40.728035979 +0000 UTC m=+3.138346063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs") pod "network-metrics-daemon-ndrsb" (UID: "42d754f9-b01a-4e28-822e-79349dec9556") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-kubelet-config\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-cni-binary-copy\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-ovnkube-config\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjz9\" (UniqueName: \"kubernetes.io/projected/b1106185-4212-4823-9a16-799768b162ff-kube-api-access-2vjz9\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-var-lib-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-netns\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-sys-fs\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8e9951e-e709-431d-9017-10e337c05160-iptables-alerter-script\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-hostroot\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-cni-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-cni-netd\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.228427 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-etc-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-netns\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-socket-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-hostroot\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-run-multus-certs\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cef15672-783f-42d2-a325-88f90a658e96-konnectivity-ca\") pod \"konnectivity-agent-xv525\" (UID: \"cef15672-783f-42d2-a325-88f90a658e96\") " pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-etc-openvswitch\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-device-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-device-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-env-overrides\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-system-cni-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-socket-dir-parent\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-etc-kubernetes\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-system-cni-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3337f71d-84e5-4047-b6e3-614c0143b182-serviceca\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbn8f\" (UniqueName: \"kubernetes.io/projected/a8e9951e-e709-431d-9017-10e337c05160-kube-api-access-xbn8f\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-socket-dir-parent\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-etc-kubernetes\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.229164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-registration-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31debb12-8b51-43d8-8c51-4bba3cd84bcd-tmp-dir\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-node-log\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3337f71d-84e5-4047-b6e3-614c0143b182-host\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-942gj\" (UniqueName: \"kubernetes.io/projected/3337f71d-84e5-4047-b6e3-614c0143b182-kube-api-access-942gj\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-cni-bin\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-env-overrides\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31debb12-8b51-43d8-8c51-4bba3cd84bcd-hosts-file\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-cnibin\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.228999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-cni-multus\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-daemon-config\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x47s\" (UniqueName: \"kubernetes.io/projected/798748c6-4900-47af-b91d-12f8468fb97d-kube-api-access-7x47s\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-kubelet\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cef15672-783f-42d2-a325-88f90a658e96-konnectivity-ca\") pod \"konnectivity-agent-xv525\" (UID: \"cef15672-783f-42d2-a325-88f90a658e96\") " pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-ovn\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1106185-4212-4823-9a16-799768b162ff-ovnkube-script-lib\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-cni-bin\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3337f71d-84e5-4047-b6e3-614c0143b182-host\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3337f71d-84e5-4047-b6e3-614c0143b182-serviceca\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-cni-bin\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229219    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-node-log\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-cni-bin\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-kubelet\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-conf-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.229338    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cef15672-783f-42d2-a325-88f90a658e96-agent-certs\") pod \"konnectivity-agent-xv525\" (UID: \"cef15672-783f-42d2-a325-88f90a658e96\") " pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.229401    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret podName:c98280f8-b5ea-4ad5-b872-7c3d9f9379dd nodeName:}" failed. No retries permitted until 2026-09-08 18:23:40.729383543 +0000 UTC m=+3.139693626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret") pod "global-pull-secret-syncer-m824h" (UID: "c98280f8-b5ea-4ad5-b872-7c3d9f9379dd") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31debb12-8b51-43d8-8c51-4bba3cd84bcd-tmp-dir\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-systemd-units\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.230724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8e9951e-e709-431d-9017-10e337c05160-host-slash\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-cni-multus\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-kubelet\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31debb12-8b51-43d8-8c51-4bba3cd84bcd-hosts-file\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-run-ovn\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-systemd-units\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-conf-dir\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8e9951e-e709-431d-9017-10e337c05160-host-slash\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-host-var-lib-kubelet\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1106185-4212-4823-9a16-799768b162ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-multus-daemon-config\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798748c6-4900-47af-b91d-12f8468fb97d-registration-dir\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.231340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.229747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-cnibin\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.232666 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.232635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cef15672-783f-42d2-a325-88f90a658e96-agent-certs\") pod \"konnectivity-agent-xv525\" (UID: \"cef15672-783f-42d2-a325-88f90a658e96\") " pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:23:40.232882 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.232861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1106185-4212-4823-9a16-799768b162ff-ovn-node-metrics-cert\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.238591 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.238528    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqpql\" (UniqueName: \"kubernetes.io/projected/42d754f9-b01a-4e28-822e-79349dec9556-kube-api-access-lqpql\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:40.241610 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.241559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdh22\" (UniqueName: \"kubernetes.io/projected/31debb12-8b51-43d8-8c51-4bba3cd84bcd-kube-api-access-wdh22\") pod \"node-resolver-lgmvk\" (UID: \"31debb12-8b51-43d8-8c51-4bba3cd84bcd\") " pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.241610 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.241602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-942gj\" (UniqueName: \"kubernetes.io/projected/3337f71d-84e5-4047-b6e3-614c0143b182-kube-api-access-942gj\") pod \"node-ca-gb8zd\" (UID: \"3337f71d-84e5-4047-b6e3-614c0143b182\") " pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.241941 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.241916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x47s\" (UniqueName: \"kubernetes.io/projected/798748c6-4900-47af-b91d-12f8468fb97d-kube-api-access-7x47s\") pod \"aws-ebs-csi-driver-node-5fs9k\" (UID: \"798748c6-4900-47af-b91d-12f8468fb97d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.242975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.242947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmqk\" (UniqueName: \"kubernetes.io/projected/de1d37a6-9f03-44aa-be2f-ed48fbe1b40e-kube-api-access-nqmqk\") pod \"multus-29bnw\" (UID: \"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e\") " pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.243329 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.243305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjz9\" (UniqueName: \"kubernetes.io/projected/b1106185-4212-4823-9a16-799768b162ff-kube-api-access-2vjz9\") pod \"ovnkube-node-wvdhn\" (UID: \"b1106185-4212-4823-9a16-799768b162ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.244400 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.244378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbn8f\" (UniqueName: \"kubernetes.io/projected/a8e9951e-e709-431d-9017-10e337c05160-kube-api-access-xbn8f\") pod \"iptables-alerter-rv7d7\" (UID: \"a8e9951e-e709-431d-9017-10e337c05160\") " pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.322054 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.322011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cvhnn"
Sep 08 18:23:40.337229 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.337201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pmgnl"
Sep 08 18:23:40.338849 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:40.338824    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852cc4a9_da83_4816_a835_80bd44ada77f.slice/crio-5f8669776cabdfaca8ec73670c1bcd6f9a68df86844d42c1f1c20b5d899f45d9 WatchSource:0}: Error finding container 5f8669776cabdfaca8ec73670c1bcd6f9a68df86844d42c1f1c20b5d899f45d9: Status 404 returned error can't find the container with id 5f8669776cabdfaca8ec73670c1bcd6f9a68df86844d42c1f1c20b5d899f45d9
Sep 08 18:23:40.349317 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:40.349288    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad692771_2e44_445c_b28d_1e0b7ffedaac.slice/crio-c85a1da84336689e7b2be5e94e6c7258e4c9f3c7581fa0dc7f9f313eaff0811e WatchSource:0}: Error finding container c85a1da84336689e7b2be5e94e6c7258e4c9f3c7581fa0dc7f9f313eaff0811e: Status 404 returned error can't find the container with id c85a1da84336689e7b2be5e94e6c7258e4c9f3c7581fa0dc7f9f313eaff0811e
Sep 08 18:23:40.351726 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.351708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lgmvk"
Sep 08 18:23:40.359373 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.359207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gb8zd"
Sep 08 18:23:40.362968 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:40.362882    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31debb12_8b51_43d8_8c51_4bba3cd84bcd.slice/crio-4011844a5b0fdbb27f9a8db348bc2a048514db63ee38864d096d5290fa89be15 WatchSource:0}: Error finding container 4011844a5b0fdbb27f9a8db348bc2a048514db63ee38864d096d5290fa89be15: Status 404 returned error can't find the container with id 4011844a5b0fdbb27f9a8db348bc2a048514db63ee38864d096d5290fa89be15
Sep 08 18:23:40.369311 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:40.369274    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3337f71d_84e5_4047_b6e3_614c0143b182.slice/crio-72e8aa5dc8809867f52e2aa5d6b674efbbb0ddb4315e9638f6a3979f2f3b0272 WatchSource:0}: Error finding container 72e8aa5dc8809867f52e2aa5d6b674efbbb0ddb4315e9638f6a3979f2f3b0272: Status 404 returned error can't find the container with id 72e8aa5dc8809867f52e2aa5d6b674efbbb0ddb4315e9638f6a3979f2f3b0272
Sep 08 18:23:40.370051 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.370035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rv7d7"
Sep 08 18:23:40.379150 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:40.379124    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e9951e_e709_431d_9017_10e337c05160.slice/crio-f1f6ecb2d736da31df990f0fffd22d7a6304413f4528a55556e4a7bd9a1a875e WatchSource:0}: Error finding container f1f6ecb2d736da31df990f0fffd22d7a6304413f4528a55556e4a7bd9a1a875e: Status 404 returned error can't find the container with id f1f6ecb2d736da31df990f0fffd22d7a6304413f4528a55556e4a7bd9a1a875e
Sep 08 18:23:40.379247 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.379235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:23:40.388168 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.388143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:23:40.397286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.397265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k"
Sep 08 18:23:40.404657 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.404637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29bnw"
Sep 08 18:23:40.449906 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.449877    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:40.632288 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.632188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:40.632483 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.632340    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:40.632483 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.632361    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:40.632483 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.632371    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2t2p9 for pod openshift-network-diagnostics/network-check-target-tgnbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:40.632483 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.632434    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9 podName:24ca9f64-7590-47f3-8925-43b6f28056e2 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:41.632415279 +0000 UTC m=+4.042725340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2t2p9" (UniqueName: "kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9") pod "network-check-target-tgnbz" (UID: "24ca9f64-7590-47f3-8925-43b6f28056e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:40.733382 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.733347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:40.733565 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:40.733396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:40.733565 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.733538    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:40.733565 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.733543    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:40.733755 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.733598    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret podName:c98280f8-b5ea-4ad5-b872-7c3d9f9379dd nodeName:}" failed. No retries permitted until 2026-09-08 18:23:41.733585152 +0000 UTC m=+4.143895227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret") pod "global-pull-secret-syncer-m824h" (UID: "c98280f8-b5ea-4ad5-b872-7c3d9f9379dd") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:40.733755 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:40.733613    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs podName:42d754f9-b01a-4e28-822e-79349dec9556 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:41.733606291 +0000 UTC m=+4.143916353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs") pod "network-metrics-daemon-ndrsb" (UID: "42d754f9-b01a-4e28-822e-79349dec9556") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:41.070331 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.070246    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:18:41 +0000 UTC" deadline="2028-06-05 04:29:02.008380668 +0000 UTC"
Sep 08 18:23:41.070331 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.070275    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15250h5m20.938108604s"
Sep 08 18:23:41.114809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.114770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:41.114976 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.114882    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:41.128809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.128777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rv7d7" event={"ID":"a8e9951e-e709-431d-9017-10e337c05160","Type":"ContainerStarted","Data":"f1f6ecb2d736da31df990f0fffd22d7a6304413f4528a55556e4a7bd9a1a875e"}
Sep 08 18:23:41.130574 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.130508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gb8zd" event={"ID":"3337f71d-84e5-4047-b6e3-614c0143b182","Type":"ContainerStarted","Data":"72e8aa5dc8809867f52e2aa5d6b674efbbb0ddb4315e9638f6a3979f2f3b0272"}
Sep 08 18:23:41.131764 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.131723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lgmvk" event={"ID":"31debb12-8b51-43d8-8c51-4bba3cd84bcd","Type":"ContainerStarted","Data":"4011844a5b0fdbb27f9a8db348bc2a048514db63ee38864d096d5290fa89be15"}
Sep 08 18:23:41.133364 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.133337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pmgnl" event={"ID":"ad692771-2e44-445c-b28d-1e0b7ffedaac","Type":"ContainerStarted","Data":"c85a1da84336689e7b2be5e94e6c7258e4c9f3c7581fa0dc7f9f313eaff0811e"}
Sep 08 18:23:41.134890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.134864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" event={"ID":"852cc4a9-da83-4816-a835-80bd44ada77f","Type":"ContainerStarted","Data":"5f8669776cabdfaca8ec73670c1bcd6f9a68df86844d42c1f1c20b5d899f45d9"}
Sep 08 18:23:41.641931 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.641254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:41.641931 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.641506    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:41.641931 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.641531    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:41.641931 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.641545    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2t2p9 for pod openshift-network-diagnostics/network-check-target-tgnbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:41.641931 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.641608    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9 podName:24ca9f64-7590-47f3-8925-43b6f28056e2 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:43.641589169 +0000 UTC m=+6.051899234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2t2p9" (UniqueName: "kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9") pod "network-check-target-tgnbz" (UID: "24ca9f64-7590-47f3-8925-43b6f28056e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:41.743426 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.742629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:41.743426 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:41.742699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:41.743426 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.742835    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:41.743426 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.742901    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret podName:c98280f8-b5ea-4ad5-b872-7c3d9f9379dd nodeName:}" failed. No retries permitted until 2026-09-08 18:23:43.742882476 +0000 UTC m=+6.153192559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret") pod "global-pull-secret-syncer-m824h" (UID: "c98280f8-b5ea-4ad5-b872-7c3d9f9379dd") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:41.743426 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.743329    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:41.743426 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:41.743386    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs podName:42d754f9-b01a-4e28-822e-79349dec9556 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:43.743369688 +0000 UTC m=+6.153679765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs") pod "network-metrics-daemon-ndrsb" (UID: "42d754f9-b01a-4e28-822e-79349dec9556") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:41.846271 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:41.846241    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef15672_783f_42d2_a325_88f90a658e96.slice/crio-0229cb20dc3ef2117aff57a137fc41e6ae59d70c3f2baf52856a784a7fc060f8 WatchSource:0}: Error finding container 0229cb20dc3ef2117aff57a137fc41e6ae59d70c3f2baf52856a784a7fc060f8: Status 404 returned error can't find the container with id 0229cb20dc3ef2117aff57a137fc41e6ae59d70c3f2baf52856a784a7fc060f8
Sep 08 18:23:41.866650 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:23:41.866268    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798748c6_4900_47af_b91d_12f8468fb97d.slice/crio-0003b5272f8803e275c1961738e92af3ab07c3f5579a531e4b41536451ea27c3 WatchSource:0}: Error finding container 0003b5272f8803e275c1961738e92af3ab07c3f5579a531e4b41536451ea27c3: Status 404 returned error can't find the container with id 0003b5272f8803e275c1961738e92af3ab07c3f5579a531e4b41536451ea27c3
Sep 08 18:23:42.070985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.070865    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:18:41 +0000 UTC" deadline="2028-05-18 19:56:09.500236264 +0000 UTC"
Sep 08 18:23:42.070985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.070984    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14833h32m27.429257177s"
Sep 08 18:23:42.125729 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.125701    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:42.125907 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:42.125826    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:42.126243 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.126224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:42.126357 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:42.126335    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:42.141325 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.141289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k" event={"ID":"798748c6-4900-47af-b91d-12f8468fb97d","Type":"ContainerStarted","Data":"0003b5272f8803e275c1961738e92af3ab07c3f5579a531e4b41536451ea27c3"}
Sep 08 18:23:42.145893 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.145747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29bnw" event={"ID":"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e","Type":"ContainerStarted","Data":"826e718df63111e154ce182d7dbfeff2ac16edde233a875f2271e84e21c4e6ce"}
Sep 08 18:23:42.149693 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.149662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"810dbe32ebae48713d661fd32a2029aed640acbb21692031822eea4f09151db2"}
Sep 08 18:23:42.153957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.153906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xv525" event={"ID":"cef15672-783f-42d2-a325-88f90a658e96","Type":"ContainerStarted","Data":"0229cb20dc3ef2117aff57a137fc41e6ae59d70c3f2baf52856a784a7fc060f8"}
Sep 08 18:23:42.159326 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:42.159288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" event={"ID":"19d2e81f86efd718848a5fc68affb80f","Type":"ContainerStarted","Data":"0b96fd3d61dbc7c255f1bc30e8011e9550aa05e72e99aca0fb22f7c019a94994"}
Sep 08 18:23:43.114724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:43.114132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:43.114724 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.114274    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:43.180893 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:43.179844    2580 generic.go:356] "Generic (PLEG): container finished" podID="1bff7db94e64466d52b7d7679d6fa34c" containerID="0885a449ed3cd6d20ce8b36d384b8b526e0a71401c3604f0e9efd72fdc60128f" exitCode=0
Sep 08 18:23:43.180893 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:43.180573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" event={"ID":"1bff7db94e64466d52b7d7679d6fa34c","Type":"ContainerDied","Data":"0885a449ed3cd6d20ce8b36d384b8b526e0a71401c3604f0e9efd72fdc60128f"}
Sep 08 18:23:43.199590 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:43.199451    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" podStartSLOduration=3.199429682 podStartE2EDuration="3.199429682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:23:42.17641556 +0000 UTC m=+4.586725647" watchObservedRunningTime="2026-09-08 18:23:43.199429682 +0000 UTC m=+5.609739768"
Sep 08 18:23:43.660164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:43.660043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:43.660372 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.660228    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:43.660372 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.660248    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:43.660372 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.660262    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2t2p9 for pod openshift-network-diagnostics/network-check-target-tgnbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:43.660372 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.660329    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9 podName:24ca9f64-7590-47f3-8925-43b6f28056e2 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:47.66030834 +0000 UTC m=+10.070618400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2t2p9" (UniqueName: "kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9") pod "network-check-target-tgnbz" (UID: "24ca9f64-7590-47f3-8925-43b6f28056e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:43.761296 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:43.761256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:43.761464 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:43.761327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:43.761464 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.761424    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:43.761464 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.761446    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:43.761677 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.761511    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs podName:42d754f9-b01a-4e28-822e-79349dec9556 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:47.761489657 +0000 UTC m=+10.171799740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs") pod "network-metrics-daemon-ndrsb" (UID: "42d754f9-b01a-4e28-822e-79349dec9556") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:43.761677 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:43.761533    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret podName:c98280f8-b5ea-4ad5-b872-7c3d9f9379dd nodeName:}" failed. No retries permitted until 2026-09-08 18:23:47.761523422 +0000 UTC m=+10.171833483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret") pod "global-pull-secret-syncer-m824h" (UID: "c98280f8-b5ea-4ad5-b872-7c3d9f9379dd") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:44.119539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:44.119440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:44.120059 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:44.119440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:44.120059 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:44.119603    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:44.120059 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:44.119693    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:45.115953 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:45.115272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:45.115953 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:45.115481    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:46.115535 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:46.114995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:46.115535 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:46.115104    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:46.116073 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:46.115733    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:46.116073 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:46.115873    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:47.114229 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:47.114195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:47.114414 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.114327    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:47.699060 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:47.699017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:47.699567 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.699219    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:47.699567 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.699244    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:47.699567 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.699255    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2t2p9 for pod openshift-network-diagnostics/network-check-target-tgnbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:47.699567 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.699308    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9 podName:24ca9f64-7590-47f3-8925-43b6f28056e2 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:55.699293615 +0000 UTC m=+18.109603675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2t2p9" (UniqueName: "kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9") pod "network-check-target-tgnbz" (UID: "24ca9f64-7590-47f3-8925-43b6f28056e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:47.800471 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:47.800425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:47.800655 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:47.800482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:47.800655 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.800587    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:47.800655 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.800612    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:47.800655 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.800643    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret podName:c98280f8-b5ea-4ad5-b872-7c3d9f9379dd nodeName:}" failed. No retries permitted until 2026-09-08 18:23:55.800627873 +0000 UTC m=+18.210937948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret") pod "global-pull-secret-syncer-m824h" (UID: "c98280f8-b5ea-4ad5-b872-7c3d9f9379dd") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:47.800820 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:47.800668    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs podName:42d754f9-b01a-4e28-822e-79349dec9556 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:55.800653286 +0000 UTC m=+18.210963346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs") pod "network-metrics-daemon-ndrsb" (UID: "42d754f9-b01a-4e28-822e-79349dec9556") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:48.117965 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:48.117725    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:48.117965 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:48.117867    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:48.117965 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:48.117927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:48.118251 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:48.118020    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:49.115233 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:49.115185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:49.115741 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:49.115345    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:50.114179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:50.114140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:50.114376 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:50.114293    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:50.114376 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:50.114351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:50.114538 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:50.114499    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:51.115300 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:51.114780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:51.115300 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:51.114920    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:52.114904 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:52.114817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:52.115124 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:52.115096    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:52.115714 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:52.115690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:52.116131 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:52.115876    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:53.114972 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:53.114935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:53.115127 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:53.115048    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:54.114896 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:54.114863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:54.115334 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:54.115011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:54.115334 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:54.115071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:54.115334 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:54.115205    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:55.114266 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:55.114228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:55.114483 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.114355    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:55.761217 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:55.761178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:55.761673 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.761377    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:55.761673 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.761405    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:55.761673 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.761420    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2t2p9 for pod openshift-network-diagnostics/network-check-target-tgnbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:55.761673 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.761506    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9 podName:24ca9f64-7590-47f3-8925-43b6f28056e2 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:11.761488244 +0000 UTC m=+34.171798306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2t2p9" (UniqueName: "kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9") pod "network-check-target-tgnbz" (UID: "24ca9f64-7590-47f3-8925-43b6f28056e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:55.861703 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:55.861667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:55.861862 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:55.861749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:55.861862 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.861822    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:55.861862 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.861840    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:55.861986 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.861895    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs podName:42d754f9-b01a-4e28-822e-79349dec9556 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:11.861876088 +0000 UTC m=+34.272186169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs") pod "network-metrics-daemon-ndrsb" (UID: "42d754f9-b01a-4e28-822e-79349dec9556") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:55.861986 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:55.861913    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret podName:c98280f8-b5ea-4ad5-b872-7c3d9f9379dd nodeName:}" failed. No retries permitted until 2026-09-08 18:24:11.861903628 +0000 UTC m=+34.272213701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret") pod "global-pull-secret-syncer-m824h" (UID: "c98280f8-b5ea-4ad5-b872-7c3d9f9379dd") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:56.114781 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:56.114754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:56.114948 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:56.114887    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:56.115052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:56.114754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:56.115155 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:56.115130    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:57.114752 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:57.114560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:57.115217 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:57.114830    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:23:58.114824 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:58.114790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:23:58.115292 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:58.114894    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:23:58.115292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:58.115004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:23:58.115292 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:58.115141    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:23:59.114176 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:23:59.114136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:23:59.114370 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:23:59.114264    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:24:00.114554 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.114196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:00.114554 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:00.114333    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:24:00.115111 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.114596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:00.115111 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:00.114684    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:24:00.218979 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.218729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lgmvk" event={"ID":"31debb12-8b51-43d8-8c51-4bba3cd84bcd","Type":"ContainerStarted","Data":"096914dd6becdb8c9f12eb280f043632162c71ed362ae38a34c3dc06fa0cee80"}
Sep 08 18:24:00.220300 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.220278    2580 generic.go:356] "Generic (PLEG): container finished" podID="852cc4a9-da83-4816-a835-80bd44ada77f" containerID="fb5e66cd1d1e800c48eb83b5513c174b2dd5c574269ba8ab9d17de9a12452944" exitCode=0
Sep 08 18:24:00.220382 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.220361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" event={"ID":"852cc4a9-da83-4816-a835-80bd44ada77f","Type":"ContainerDied","Data":"fb5e66cd1d1e800c48eb83b5513c174b2dd5c574269ba8ab9d17de9a12452944"}
Sep 08 18:24:00.222183 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.222164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k" event={"ID":"798748c6-4900-47af-b91d-12f8468fb97d","Type":"ContainerStarted","Data":"8d0ee5a5db9b2729604d9e613ba8fa8fb4664e399420629b5a2fb7b1eb672ad3"}
Sep 08 18:24:00.224225 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.224200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29bnw" event={"ID":"de1d37a6-9f03-44aa-be2f-ed48fbe1b40e","Type":"ContainerStarted","Data":"f27564caa958b579480b0798cc2186c9343c7cffb74d4cf58813cd3dd6a9b65b"}
Sep 08 18:24:00.227483 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.227404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"d0735817ef986cd241959720e56067d13b55ebaaecae92a042f7b73c6f3e93d1"}
Sep 08 18:24:00.227571 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.227505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"4007e31bfc776380dd34f1150be3d205f3edbe1238bdb871f92bba9e530aec31"}
Sep 08 18:24:00.227571 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.227554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"d89a5b2ce49677905ef9da3f393c59ca7a5db3f6fe8b01aa228cc11a12da3c31"}
Sep 08 18:24:00.229459 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.228975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xv525" event={"ID":"cef15672-783f-42d2-a325-88f90a658e96","Type":"ContainerStarted","Data":"bb01e498bff0e9290bdc5668c2253faf5f520335a8d175993b3995a76dc3ab1e"}
Sep 08 18:24:00.231248 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.231231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" event={"ID":"1bff7db94e64466d52b7d7679d6fa34c","Type":"ContainerStarted","Data":"e9797904f114012a80c0aab4d271f61f05a9b7c2cc9310895a4b07109d1535b4"}
Sep 08 18:24:00.233105 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.233077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gb8zd" event={"ID":"3337f71d-84e5-4047-b6e3-614c0143b182","Type":"ContainerStarted","Data":"640c35484f97acfd14287e2fddf63c3025ce4e93e89470e544c02d5260060361"}
Sep 08 18:24:00.235022 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.234996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pmgnl" event={"ID":"ad692771-2e44-445c-b28d-1e0b7ffedaac","Type":"ContainerStarted","Data":"7b7ab9dc895203c17ab9a52ef8e9075f910ad9e4d5be363a4f0464aa449e9ba7"}
Sep 08 18:24:00.268480 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.268397    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-29bnw" podStartSLOduration=3.135521922 podStartE2EDuration="21.268381541s" totalImagesPullingTime="18.132859619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:00.267976779 +0000 UTC m=+22.678286860" watchObservedRunningTime="2026-09-08 18:24:00.268381541 +0000 UTC m=+22.678691621"
Sep 08 18:24:00.285136 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.285093    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" podStartSLOduration=20.285077488 podStartE2EDuration="20.285077488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:00.284526546 +0000 UTC m=+22.694836640" watchObservedRunningTime="2026-09-08 18:24:00.285077488 +0000 UTC m=+22.695387570"
Sep 08 18:24:00.323363 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.323316    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xv525" podStartSLOduration=7.4920800100000005 podStartE2EDuration="21.323300509s" totalImagesPullingTime="13.831220499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:00.323297687 +0000 UTC m=+22.733607767" watchObservedRunningTime="2026-09-08 18:24:00.323300509 +0000 UTC m=+22.733610589"
Sep 08 18:24:00.323520 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.323396    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pmgnl" podStartSLOduration=1.8886917699999999 podStartE2EDuration="21.323390521s" totalImagesPullingTime="19.434698751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:00.303603826 +0000 UTC m=+22.713913898" watchObservedRunningTime="2026-09-08 18:24:00.323390521 +0000 UTC m=+22.733700601"
Sep 08 18:24:00.356021 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.355963    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lgmvk" podStartSLOduration=1.9350176000000001 podStartE2EDuration="21.355943792s" totalImagesPullingTime="19.420926192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:00.340384844 +0000 UTC m=+22.750694921" watchObservedRunningTime="2026-09-08 18:24:00.355943792 +0000 UTC m=+22.766253872"
Sep 08 18:24:00.356157 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:00.356062    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gb8zd" podStartSLOduration=9.822428069 podStartE2EDuration="21.356055253s" totalImagesPullingTime="11.533627184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:00.355718954 +0000 UTC m=+22.766029037" watchObservedRunningTime="2026-09-08 18:24:00.356055253 +0000 UTC m=+22.766365339"
Sep 08 18:24:01.114890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:01.114859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:01.115638 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:01.114986    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:24:01.241812 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:01.241768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"9930b663f52cab77e8eef3e31a89b87053e9918df3b232b60afd27108c7f749e"}
Sep 08 18:24:01.241995 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:01.241846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"117863bc301c2d685d56ee2b8fc57e285d5cc27c190f734024eb9347c7a07ba0"}
Sep 08 18:24:01.241995 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:01.241894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"90ad4a223d2a81977b6646ee7533e27d8841d2f43229f25ec251bd07e4ccdae6"}
Sep 08 18:24:01.403666 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:01.403642    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 18:24:01.732098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:01.732014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:24:01.732587 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:01.732563    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:24:02.089130 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.088987    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T18:24:01.403661105Z","UUID":"ecd0cf18-aa28-47de-ba98-95a13d91994e","Handler":null,"Name":"","Endpoint":""}
Sep 08 18:24:02.090874 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.090846    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 18:24:02.090999 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.090881    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 18:24:02.114817 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.114790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:02.114817 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.114814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:02.115454 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:02.114919    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:24:02.115454 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:02.115053    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:24:02.247015 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.246970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k" event={"ID":"798748c6-4900-47af-b91d-12f8468fb97d","Type":"ContainerStarted","Data":"32f140655c787b9c94e426574b212b236ff1313556e4072923740631c5f5bed5"}
Sep 08 18:24:02.248570 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.248539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rv7d7" event={"ID":"a8e9951e-e709-431d-9017-10e337c05160","Type":"ContainerStarted","Data":"e1068e2ab123aab8e9e274d630bddf0cb63912f05d8949e9a0b0c2ba1f452c1d"}
Sep 08 18:24:02.248907 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.248884    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:24:02.249527 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.249505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xv525"
Sep 08 18:24:02.264734 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:02.264679    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rv7d7" podStartSLOduration=7.965187996 podStartE2EDuration="23.264658943s" totalImagesPullingTime="15.299470947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:02.263845301 +0000 UTC m=+24.674155382" watchObservedRunningTime="2026-09-08 18:24:02.264658943 +0000 UTC m=+24.674969029"
Sep 08 18:24:03.114693 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:03.114493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:03.114848 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:03.114811    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:24:03.254974 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:03.254895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"6517989f75fbf08f3c21354557f06efd3b583d7c50589b4eec6748dd014841d0"}
Sep 08 18:24:03.257239 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:03.257209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k" event={"ID":"798748c6-4900-47af-b91d-12f8468fb97d","Type":"ContainerStarted","Data":"c5885e8143a7ce49d02e34a6a3906e9c54be8dcbc6cd6779f0460924a3253b41"}
Sep 08 18:24:03.274856 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:03.274811    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fs9k" podStartSLOduration=7.837647421 podStartE2EDuration="24.274796488s" totalImagesPullingTime="16.437149067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:03.274393695 +0000 UTC m=+25.684703777" watchObservedRunningTime="2026-09-08 18:24:03.274796488 +0000 UTC m=+25.685106612"
Sep 08 18:24:04.117679 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:04.117649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:04.117886 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:04.117651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:04.117886 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:04.117778    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:24:04.117975 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:04.117888    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:24:05.114180 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:05.114103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:05.114736 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:05.114237    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:24:05.265133 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:05.264970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" event={"ID":"b1106185-4212-4823-9a16-799768b162ff","Type":"ContainerStarted","Data":"ebb2c8bc75999cffc7ec7b048349519e146172289361fb7412f8f4bc959de55f"}
Sep 08 18:24:05.265373 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:05.265357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:24:05.265465 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:05.265452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:24:05.281860 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:05.281799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:24:05.295954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:05.295887    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn" podStartSLOduration=8.334215785 podStartE2EDuration="26.295873281s" totalImagesPullingTime="17.961657496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:05.295519529 +0000 UTC m=+27.705829603" watchObservedRunningTime="2026-09-08 18:24:05.295873281 +0000 UTC m=+27.706183361"
Sep 08 18:24:06.116926 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:06.116897    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:06.117321 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:06.116998    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:24:06.117321 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:06.117052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:06.117321 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:06.117132    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:24:06.272582 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:06.272542    2580 generic.go:356] "Generic (PLEG): container finished" podID="852cc4a9-da83-4816-a835-80bd44ada77f" containerID="9a8c3b7845f7ab5fec1261ad6f40d4864b6884411f4883f2eda9735ef827107c" exitCode=0
Sep 08 18:24:06.273251 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:06.273217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" event={"ID":"852cc4a9-da83-4816-a835-80bd44ada77f","Type":"ContainerDied","Data":"9a8c3b7845f7ab5fec1261ad6f40d4864b6884411f4883f2eda9735ef827107c"}
Sep 08 18:24:06.273906 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:06.273890    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:24:06.292417 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:06.292087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:24:07.115274 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.115252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:07.115399 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:07.115346    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:24:07.135086 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.135056    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m824h"]
Sep 08 18:24:07.135990 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.135164    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:07.135990 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:07.135265    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:24:07.140181 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.140154    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgnbz"]
Sep 08 18:24:07.140701 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.140680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ndrsb"]
Sep 08 18:24:07.140798 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.140763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:07.140874 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:07.140857    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:24:07.276954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.276923    2580 generic.go:356] "Generic (PLEG): container finished" podID="852cc4a9-da83-4816-a835-80bd44ada77f" containerID="0dc156e7b8547622373292d7d0ff70901470e3ed3ac51baee7970008bbf22531" exitCode=0
Sep 08 18:24:07.277098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.277001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:07.277098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:07.277003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" event={"ID":"852cc4a9-da83-4816-a835-80bd44ada77f","Type":"ContainerDied","Data":"0dc156e7b8547622373292d7d0ff70901470e3ed3ac51baee7970008bbf22531"}
Sep 08 18:24:07.277194 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:07.277137    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:24:09.115303 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:09.115109    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:09.115674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:09.115163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:09.115674 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:09.115362    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:24:09.115674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:09.115172    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:09.115674 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:09.115414    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:24:09.115674 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:09.115549    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:24:09.283777 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:09.283747    2580 generic.go:356] "Generic (PLEG): container finished" podID="852cc4a9-da83-4816-a835-80bd44ada77f" containerID="dd06d9f495ac2871db24444146508ee9fb94ea6620910b2ed5f7780fa869547d" exitCode=0
Sep 08 18:24:09.283936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:09.283807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" event={"ID":"852cc4a9-da83-4816-a835-80bd44ada77f","Type":"ContainerDied","Data":"dd06d9f495ac2871db24444146508ee9fb94ea6620910b2ed5f7780fa869547d"}
Sep 08 18:24:11.114835 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:11.114799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:11.115290 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:11.114801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:11.115290 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.114941    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndrsb" podUID="42d754f9-b01a-4e28-822e-79349dec9556"
Sep 08 18:24:11.115290 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:11.114812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:11.115290 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.114984    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgnbz" podUID="24ca9f64-7590-47f3-8925-43b6f28056e2"
Sep 08 18:24:11.115290 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.115052    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m824h" podUID="c98280f8-b5ea-4ad5-b872-7c3d9f9379dd"
Sep 08 18:24:11.781488 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:11.781456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:11.781773 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.781591    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:24:11.781773 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.781614    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:24:11.781773 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.781628    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2t2p9 for pod openshift-network-diagnostics/network-check-target-tgnbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:24:11.781773 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.781692    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9 podName:24ca9f64-7590-47f3-8925-43b6f28056e2 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:43.781672701 +0000 UTC m=+66.191982774 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2t2p9" (UniqueName: "kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9") pod "network-check-target-tgnbz" (UID: "24ca9f64-7590-47f3-8925-43b6f28056e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:24:11.882756 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:11.882719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:11.882897 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:11.882782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:11.882897 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.882873    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:24:11.882897 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.882873    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:24:11.883037 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.882932    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret podName:c98280f8-b5ea-4ad5-b872-7c3d9f9379dd nodeName:}" failed. No retries permitted until 2026-09-08 18:24:43.882916495 +0000 UTC m=+66.293226570 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret") pod "global-pull-secret-syncer-m824h" (UID: "c98280f8-b5ea-4ad5-b872-7c3d9f9379dd") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:24:11.883037 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:11.882950    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs podName:42d754f9-b01a-4e28-822e-79349dec9556 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:43.882940954 +0000 UTC m=+66.293251018 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs") pod "network-metrics-daemon-ndrsb" (UID: "42d754f9-b01a-4e28-822e-79349dec9556") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:24:12.890875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.890846    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeReady"
Sep 08 18:24:12.891283 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.890987    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 18:24:12.933682 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.933649    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-44lg7"]
Sep 08 18:24:12.938476 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.938422    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7gf7v"]
Sep 08 18:24:12.938626 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.938558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:12.941222 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.941188    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 18:24:12.941350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.941202    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 18:24:12.941412 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.941399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vbqmk\""
Sep 08 18:24:12.942087 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.942069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7gf7v"
Sep 08 18:24:12.944770 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.944575    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jndxk\""
Sep 08 18:24:12.944770 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.944640    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 18:24:12.944770 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.944673    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 18:24:12.944770 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.944766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 18:24:12.946935 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.946903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-44lg7"]
Sep 08 18:24:12.949657 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:12.949628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7gf7v"]
Sep 08 18:24:13.092630 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.092536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-tmp-dir\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.092630 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.092601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kl9\" (UniqueName: \"kubernetes.io/projected/97d26502-e9d6-4c71-92e3-46f65d798522-kube-api-access-v6kl9\") pod \"ingress-canary-7gf7v\" (UID: \"97d26502-e9d6-4c71-92e3-46f65d798522\") " pod="openshift-ingress-canary/ingress-canary-7gf7v"
Sep 08 18:24:13.092849 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.092691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d26502-e9d6-4c71-92e3-46f65d798522-cert\") pod \"ingress-canary-7gf7v\" (UID: \"97d26502-e9d6-4c71-92e3-46f65d798522\") " pod="openshift-ingress-canary/ingress-canary-7gf7v"
Sep 08 18:24:13.092849 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.092732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-config-volume\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.092849 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.092771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.092849 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.092798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65j8m\" (UniqueName: \"kubernetes.io/projected/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-kube-api-access-65j8m\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.114329 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.114284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:13.114533 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.114333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:13.114533 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.114333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:13.117207 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.117178    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 18:24:13.117340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.117257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jmlkr\""
Sep 08 18:24:13.117340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.117280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 18:24:13.117473 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.117393    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frqcg\""
Sep 08 18:24:13.118321 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.118303    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 18:24:13.118321 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.118316    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 18:24:13.193361 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.193313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kl9\" (UniqueName: \"kubernetes.io/projected/97d26502-e9d6-4c71-92e3-46f65d798522-kube-api-access-v6kl9\") pod \"ingress-canary-7gf7v\" (UID: \"97d26502-e9d6-4c71-92e3-46f65d798522\") " pod="openshift-ingress-canary/ingress-canary-7gf7v"
Sep 08 18:24:13.193568 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.193377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d26502-e9d6-4c71-92e3-46f65d798522-cert\") pod \"ingress-canary-7gf7v\" (UID: \"97d26502-e9d6-4c71-92e3-46f65d798522\") " pod="openshift-ingress-canary/ingress-canary-7gf7v"
Sep 08 18:24:13.193568 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.193413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-config-volume\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.193568 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.193463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.193568 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.193488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65j8m\" (UniqueName: \"kubernetes.io/projected/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-kube-api-access-65j8m\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.193568 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.193538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-tmp-dir\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.193894 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:13.193832    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:13.193951 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:13.193898    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls podName:cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:13.693874141 +0000 UTC m=+36.104184218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls") pod "dns-default-44lg7" (UID: "cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:13.193951 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.193915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-tmp-dir\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.194150 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.194128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-config-volume\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.198463 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.198413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d26502-e9d6-4c71-92e3-46f65d798522-cert\") pod \"ingress-canary-7gf7v\" (UID: \"97d26502-e9d6-4c71-92e3-46f65d798522\") " pod="openshift-ingress-canary/ingress-canary-7gf7v"
Sep 08 18:24:13.202361 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.202340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65j8m\" (UniqueName: \"kubernetes.io/projected/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-kube-api-access-65j8m\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.202759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.202725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kl9\" (UniqueName: \"kubernetes.io/projected/97d26502-e9d6-4c71-92e3-46f65d798522-kube-api-access-v6kl9\") pod \"ingress-canary-7gf7v\" (UID: \"97d26502-e9d6-4c71-92e3-46f65d798522\") " pod="openshift-ingress-canary/ingress-canary-7gf7v"
Sep 08 18:24:13.259220 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.259179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7gf7v"
Sep 08 18:24:13.698113 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:13.698065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:13.698290 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:13.698232    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:13.698358 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:13.698309    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls podName:cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:14.698292521 +0000 UTC m=+37.108602584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls") pod "dns-default-44lg7" (UID: "cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:14.204591 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:14.204564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7gf7v"]
Sep 08 18:24:14.222517 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:14.222484    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d26502_e9d6_4c71_92e3_46f65d798522.slice/crio-61b00dbae0b7a3e2947b2c132467e931329e01cdbb28ae7a665bc0ba9b8a57e0 WatchSource:0}: Error finding container 61b00dbae0b7a3e2947b2c132467e931329e01cdbb28ae7a665bc0ba9b8a57e0: Status 404 returned error can't find the container with id 61b00dbae0b7a3e2947b2c132467e931329e01cdbb28ae7a665bc0ba9b8a57e0
Sep 08 18:24:14.296884 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:14.296853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7gf7v" event={"ID":"97d26502-e9d6-4c71-92e3-46f65d798522","Type":"ContainerStarted","Data":"61b00dbae0b7a3e2947b2c132467e931329e01cdbb28ae7a665bc0ba9b8a57e0"}
Sep 08 18:24:14.299474 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:14.299452    2580 generic.go:356] "Generic (PLEG): container finished" podID="852cc4a9-da83-4816-a835-80bd44ada77f" containerID="e9396fa6c27f2f9de6fda24173badb197c561b66385c518289eb505363860eb3" exitCode=0
Sep 08 18:24:14.299582 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:14.299484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" event={"ID":"852cc4a9-da83-4816-a835-80bd44ada77f","Type":"ContainerDied","Data":"e9396fa6c27f2f9de6fda24173badb197c561b66385c518289eb505363860eb3"}
Sep 08 18:24:14.704287 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:14.704251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:14.704474 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:14.704446    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:14.704526 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:14.704514    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls podName:cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:16.704499391 +0000 UTC m=+39.114809451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls") pod "dns-default-44lg7" (UID: "cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:15.305780 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:15.305744    2580 generic.go:356] "Generic (PLEG): container finished" podID="852cc4a9-da83-4816-a835-80bd44ada77f" containerID="efa132960d8352fc31982705d90adb80b64f01cbd8cf49193984c0ebf6bb38c8" exitCode=0
Sep 08 18:24:15.306223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:15.305813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" event={"ID":"852cc4a9-da83-4816-a835-80bd44ada77f","Type":"ContainerDied","Data":"efa132960d8352fc31982705d90adb80b64f01cbd8cf49193984c0ebf6bb38c8"}
Sep 08 18:24:16.312254 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:16.312220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" event={"ID":"852cc4a9-da83-4816-a835-80bd44ada77f","Type":"ContainerStarted","Data":"28631e2d1ab462e590207637113541c42849a89ce47f8f035e62e514681fd7e4"}
Sep 08 18:24:16.335346 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:16.335290    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cvhnn" podStartSLOduration=10.144508255 podStartE2EDuration="37.335271931s" totalImagesPullingTime="27.190763676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:16.333321779 +0000 UTC m=+38.743631862" watchObservedRunningTime="2026-09-08 18:24:16.335271931 +0000 UTC m=+38.745582011"
Sep 08 18:24:16.721891 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:16.721859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:16.721987 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:16.721965    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:16.722034 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:16.722025    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls podName:cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:20.722011977 +0000 UTC m=+43.132322048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls") pod "dns-default-44lg7" (UID: "cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:17.316297 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:17.316260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7gf7v" event={"ID":"97d26502-e9d6-4c71-92e3-46f65d798522","Type":"ContainerStarted","Data":"13c8b13a126af028da1802c4f85757b6c3f43df293e74d20754142ae7454891d"}
Sep 08 18:24:18.679458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.679392    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7gf7v" podStartSLOduration=4.172387332 podStartE2EDuration="6.679375832s" totalImagesPullingTime="2.5069885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:17.333600269 +0000 UTC m=+39.743910350" watchObservedRunningTime="2026-09-08 18:24:18.679375832 +0000 UTC m=+41.089685928"
Sep 08 18:24:18.679810 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.679523    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr"]
Sep 08 18:24:18.713554 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.713526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr"]
Sep 08 18:24:18.713691 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.713653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr"
Sep 08 18:24:18.717181 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.717153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-86gs6\""
Sep 08 18:24:18.717468 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.717420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:24:18.717556 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.717491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 08 18:24:18.783708 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.783683    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"]
Sep 08 18:24:18.805637 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.805610    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ckf7z"]
Sep 08 18:24:18.805765 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.805713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:18.808628 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.808611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 18:24:18.808740 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.808612    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 18:24:18.808938 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.808920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nkf9w\""
Sep 08 18:24:18.809001 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.808948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 18:24:18.809001 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.808984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:24:18.825710 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.825689    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cd48b8b66-dnpdq"]
Sep 08 18:24:18.825828 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.825813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-ckf7z"
Sep 08 18:24:18.828365 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.828343    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-sd4kj\""
Sep 08 18:24:18.835986 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.835966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dg8k\" (UniqueName: \"kubernetes.io/projected/07dc1c3a-0077-48b4-a8f7-c6700e878e85-kube-api-access-2dg8k\") pod \"volume-data-source-validator-695c895bd6-s75dr\" (UID: \"07dc1c3a-0077-48b4-a8f7-c6700e878e85\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr"
Sep 08 18:24:18.847100 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.847084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"]
Sep 08 18:24:18.847191 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.847112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cd48b8b66-dnpdq"]
Sep 08 18:24:18.847191 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.847125    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ckf7z"]
Sep 08 18:24:18.847261 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.847201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.851031 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.851015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 18:24:18.851378 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.851355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 18:24:18.851476 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.851400    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cm2c5\""
Sep 08 18:24:18.851524 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.851475    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 18:24:18.858124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.858105    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 18:24:18.885841 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.885822    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"]
Sep 08 18:24:18.907635 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.907615    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"]
Sep 08 18:24:18.907775 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.907758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:18.911029 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.911009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 08 18:24:18.911178 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.911033    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 18:24:18.911178 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.911119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8nhwz\""
Sep 08 18:24:18.911272 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.911263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 18:24:18.913132 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.913117    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 08 18:24:18.931738 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.931679    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-djllr"]
Sep 08 18:24:18.931826 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.931798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:18.935952 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.934745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 08 18:24:18.935952 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.934872    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 08 18:24:18.935952 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.934745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 08 18:24:18.935952 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.935107    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:24:18.935952 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.935135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-r88d8\""
Sep 08 18:24:18.936700 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.936665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-installation-pull-secrets\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.936810 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.936717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.936865 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.936825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-image-registry-private-configuration\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.936931 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.936872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrx87\" (UniqueName: \"kubernetes.io/projected/d79eeaa4-08bf-4e29-acfd-cf186424e109-kube-api-access-vrx87\") pod \"network-check-source-685456655f-ckf7z\" (UID: \"d79eeaa4-08bf-4e29-acfd-cf186424e109\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ckf7z"
Sep 08 18:24:18.937028 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.936985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d7b263-636a-4c20-9c72-bb56f51615f4-ca-trust-extracted\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.937028 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.937020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-certificates\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.937131 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.937071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d10402b8-2af5-478a-9663-b9efdaf54529-serving-cert\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:18.937556 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.937530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwxm\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-kube-api-access-7hwxm\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.937638 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.937576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gsh\" (UniqueName: \"kubernetes.io/projected/d10402b8-2af5-478a-9663-b9efdaf54529-kube-api-access-q2gsh\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:18.937638 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.937612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-bound-sa-token\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.937731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.937663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d10402b8-2af5-478a-9663-b9efdaf54529-config\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:18.937731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.937685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dg8k\" (UniqueName: \"kubernetes.io/projected/07dc1c3a-0077-48b4-a8f7-c6700e878e85-kube-api-access-2dg8k\") pod \"volume-data-source-validator-695c895bd6-s75dr\" (UID: \"07dc1c3a-0077-48b4-a8f7-c6700e878e85\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr"
Sep 08 18:24:18.937731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.937700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-trusted-ca\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:18.947795 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.947774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"]
Sep 08 18:24:18.947883 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.947811    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"]
Sep 08 18:24:18.947883 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.947830    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-djllr"]
Sep 08 18:24:18.947883 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.947877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:18.949695 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.949667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dg8k\" (UniqueName: \"kubernetes.io/projected/07dc1c3a-0077-48b4-a8f7-c6700e878e85-kube-api-access-2dg8k\") pod \"volume-data-source-validator-695c895bd6-s75dr\" (UID: \"07dc1c3a-0077-48b4-a8f7-c6700e878e85\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr"
Sep 08 18:24:18.950658 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.950636    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 18:24:18.950746 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.950640    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 18:24:18.950746 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:18.950720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ttsc8\""
Sep 08 18:24:19.025133 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.025100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr"
Sep 08 18:24:19.038438 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d10402b8-2af5-478a-9663-b9efdaf54529-serving-cert\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:19.038511 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwxm\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-kube-api-access-7hwxm\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.038511 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97b8\" (UniqueName: \"kubernetes.io/projected/40baa83d-6f8c-41f2-b74a-832246efe5cb-kube-api-access-j97b8\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.038587 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gsh\" (UniqueName: \"kubernetes.io/projected/d10402b8-2af5-478a-9663-b9efdaf54529-kube-api-access-q2gsh\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:19.038587 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-bound-sa-token\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.038587 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d10402b8-2af5-478a-9663-b9efdaf54529-config\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:19.038728 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-trusted-ca\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.038728 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-installation-pull-secrets\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.038728 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/40baa83d-6f8c-41f2-b74a-832246efe5cb-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.038728 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lhrf\" (UniqueName: \"kubernetes.io/projected/1bf50b32-0985-4f48-87a0-ef0b751950db-kube-api-access-6lhrf\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.038728 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-nginx-conf\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:19.038969 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.038969 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.038969 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:19.039119 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.038987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1bf50b32-0985-4f48-87a0-ef0b751950db-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.039119 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.038996    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:24:19.039119 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.039013    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd48b8b66-dnpdq: secret "image-registry-tls" not found
Sep 08 18:24:19.039119 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1bf50b32-0985-4f48-87a0-ef0b751950db-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.039119 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-image-registry-private-configuration\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.039119 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.039073    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls podName:c4d7b263-636a-4c20-9c72-bb56f51615f4 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:19.53905597 +0000 UTC m=+41.949366033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls") pod "image-registry-cd48b8b66-dnpdq" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4") : secret "image-registry-tls" not found
Sep 08 18:24:19.039402 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrx87\" (UniqueName: \"kubernetes.io/projected/d79eeaa4-08bf-4e29-acfd-cf186424e109-kube-api-access-vrx87\") pod \"network-check-source-685456655f-ckf7z\" (UID: \"d79eeaa4-08bf-4e29-acfd-cf186424e109\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ckf7z"
Sep 08 18:24:19.039402 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d7b263-636a-4c20-9c72-bb56f51615f4-ca-trust-extracted\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.039402 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-certificates\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.039402 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d10402b8-2af5-478a-9663-b9efdaf54529-config\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:19.039627 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d7b263-636a-4c20-9c72-bb56f51615f4-ca-trust-extracted\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.039730 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-certificates\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.039868 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.039790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-trusted-ca\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.040930 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.040909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d10402b8-2af5-478a-9663-b9efdaf54529-serving-cert\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:19.041328 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.041309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-image-registry-private-configuration\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.041580 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.041562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-installation-pull-secrets\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.051638 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.051618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gsh\" (UniqueName: \"kubernetes.io/projected/d10402b8-2af5-478a-9663-b9efdaf54529-kube-api-access-q2gsh\") pod \"service-ca-operator-859f94c4b-747mg\" (UID: \"d10402b8-2af5-478a-9663-b9efdaf54529\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:19.055089 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.055058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-bound-sa-token\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.055953 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.055918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrx87\" (UniqueName: \"kubernetes.io/projected/d79eeaa4-08bf-4e29-acfd-cf186424e109-kube-api-access-vrx87\") pod \"network-check-source-685456655f-ckf7z\" (UID: \"d79eeaa4-08bf-4e29-acfd-cf186424e109\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ckf7z"
Sep 08 18:24:19.056329 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.056305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwxm\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-kube-api-access-7hwxm\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.114007 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.113978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"
Sep 08 18:24:19.133518 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.133475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-ckf7z"
Sep 08 18:24:19.139865 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.139846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97b8\" (UniqueName: \"kubernetes.io/projected/40baa83d-6f8c-41f2-b74a-832246efe5cb-kube-api-access-j97b8\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.139931 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.139891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/40baa83d-6f8c-41f2-b74a-832246efe5cb-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.139931 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.139919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lhrf\" (UniqueName: \"kubernetes.io/projected/1bf50b32-0985-4f48-87a0-ef0b751950db-kube-api-access-6lhrf\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.140062 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.140041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-nginx-conf\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:19.140114 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.140103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.140161 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.140126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:19.140210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.140163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1bf50b32-0985-4f48-87a0-ef0b751950db-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.140210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.140189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1bf50b32-0985-4f48-87a0-ef0b751950db-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.140309 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.140236    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:19.140309 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.140240    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:24:19.140309 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.140303    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert podName:4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc nodeName:}" failed. No retries permitted until 2026-09-08 18:24:19.640285461 +0000 UTC m=+42.050595537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-djllr" (UID: "4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc") : secret "networking-console-plugin-cert" not found
Sep 08 18:24:19.140481 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.140317    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls podName:40baa83d-6f8c-41f2-b74a-832246efe5cb nodeName:}" failed. No retries permitted until 2026-09-08 18:24:19.640311002 +0000 UTC m=+42.050621061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f6b9v" (UID: "40baa83d-6f8c-41f2-b74a-832246efe5cb") : secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:19.140664 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.140643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/40baa83d-6f8c-41f2-b74a-832246efe5cb-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.140828 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.140807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-nginx-conf\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:19.140881 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.140824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1bf50b32-0985-4f48-87a0-ef0b751950db-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.142589 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.142572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1bf50b32-0985-4f48-87a0-ef0b751950db-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.155013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.154988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97b8\" (UniqueName: \"kubernetes.io/projected/40baa83d-6f8c-41f2-b74a-832246efe5cb-kube-api-access-j97b8\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.155091 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.155034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lhrf\" (UniqueName: \"kubernetes.io/projected/1bf50b32-0985-4f48-87a0-ef0b751950db-kube-api-access-6lhrf\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mr28f\" (UID: \"1bf50b32-0985-4f48-87a0-ef0b751950db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.241970 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.241938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"
Sep 08 18:24:19.311096 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.311055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr"]
Sep 08 18:24:19.324400 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.324199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr" event={"ID":"07dc1c3a-0077-48b4-a8f7-c6700e878e85","Type":"ContainerStarted","Data":"e37e4fb09413200dc351f722a893c90229917efc1c7d5d21275ef89e8c2d6ccb"}
Sep 08 18:24:19.411224 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.411196    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg"]
Sep 08 18:24:19.413948 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:19.413921    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10402b8_2af5_478a_9663_b9efdaf54529.slice/crio-9aa081df8858b474298e55d6cd4a1d503189c617074fbc507524eade9b23ecd1 WatchSource:0}: Error finding container 9aa081df8858b474298e55d6cd4a1d503189c617074fbc507524eade9b23ecd1: Status 404 returned error can't find the container with id 9aa081df8858b474298e55d6cd4a1d503189c617074fbc507524eade9b23ecd1
Sep 08 18:24:19.444679 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.444656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ckf7z"]
Sep 08 18:24:19.446694 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:19.446668    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79eeaa4_08bf_4e29_acfd_cf186424e109.slice/crio-54cc2c2ba06c1b7a8f7401167e0568a3e5a725e8c221091fea2d6fd350d5539f WatchSource:0}: Error finding container 54cc2c2ba06c1b7a8f7401167e0568a3e5a725e8c221091fea2d6fd350d5539f: Status 404 returned error can't find the container with id 54cc2c2ba06c1b7a8f7401167e0568a3e5a725e8c221091fea2d6fd350d5539f
Sep 08 18:24:19.543479 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.543444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:19.543622 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.543597    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:24:19.543622 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.543613    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd48b8b66-dnpdq: secret "image-registry-tls" not found
Sep 08 18:24:19.543688 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.543666    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls podName:c4d7b263-636a-4c20-9c72-bb56f51615f4 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:20.543650657 +0000 UTC m=+42.953960716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls") pod "image-registry-cd48b8b66-dnpdq" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4") : secret "image-registry-tls" not found
Sep 08 18:24:19.559940 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.559918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f"]
Sep 08 18:24:19.562593 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:19.562573    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf50b32_0985_4f48_87a0_ef0b751950db.slice/crio-79e1b25540c3a244a2bec333c4d689a8d5f65cf3777aa44676fba7bf9d902a7e WatchSource:0}: Error finding container 79e1b25540c3a244a2bec333c4d689a8d5f65cf3777aa44676fba7bf9d902a7e: Status 404 returned error can't find the container with id 79e1b25540c3a244a2bec333c4d689a8d5f65cf3777aa44676fba7bf9d902a7e
Sep 08 18:24:19.643918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.643890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:19.644040 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:19.643922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:19.644040 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.644027    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:24:19.644116 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.644034    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:19.644116 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.644074    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert podName:4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc nodeName:}" failed. No retries permitted until 2026-09-08 18:24:20.644061702 +0000 UTC m=+43.054371775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-djllr" (UID: "4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc") : secret "networking-console-plugin-cert" not found
Sep 08 18:24:19.644116 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:19.644086    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls podName:40baa83d-6f8c-41f2-b74a-832246efe5cb nodeName:}" failed. No retries permitted until 2026-09-08 18:24:20.644080186 +0000 UTC m=+43.054390245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f6b9v" (UID: "40baa83d-6f8c-41f2-b74a-832246efe5cb") : secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:20.331626 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:20.331580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-ckf7z" event={"ID":"d79eeaa4-08bf-4e29-acfd-cf186424e109","Type":"ContainerStarted","Data":"54cc2c2ba06c1b7a8f7401167e0568a3e5a725e8c221091fea2d6fd350d5539f"}
Sep 08 18:24:20.333299 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:20.333269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f" event={"ID":"1bf50b32-0985-4f48-87a0-ef0b751950db","Type":"ContainerStarted","Data":"79e1b25540c3a244a2bec333c4d689a8d5f65cf3777aa44676fba7bf9d902a7e"}
Sep 08 18:24:20.334906 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:20.334836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg" event={"ID":"d10402b8-2af5-478a-9663-b9efdaf54529","Type":"ContainerStarted","Data":"9aa081df8858b474298e55d6cd4a1d503189c617074fbc507524eade9b23ecd1"}
Sep 08 18:24:20.550307 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:20.550268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:20.550527 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.550464    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:24:20.550527 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.550487    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd48b8b66-dnpdq: secret "image-registry-tls" not found
Sep 08 18:24:20.550709 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.550550    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls podName:c4d7b263-636a-4c20-9c72-bb56f51615f4 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:22.55052904 +0000 UTC m=+44.960839112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls") pod "image-registry-cd48b8b66-dnpdq" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4") : secret "image-registry-tls" not found
Sep 08 18:24:20.652312 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:20.651297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:20.652312 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:20.651475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:20.652312 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.651608    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:20.652312 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.651686    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls podName:40baa83d-6f8c-41f2-b74a-832246efe5cb nodeName:}" failed. No retries permitted until 2026-09-08 18:24:22.651666295 +0000 UTC m=+45.061976377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f6b9v" (UID: "40baa83d-6f8c-41f2-b74a-832246efe5cb") : secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:20.652312 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.652221    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:24:20.652312 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.652275    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert podName:4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc nodeName:}" failed. No retries permitted until 2026-09-08 18:24:22.652260599 +0000 UTC m=+45.062570663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-djllr" (UID: "4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc") : secret "networking-console-plugin-cert" not found
Sep 08 18:24:20.752097 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:20.752056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:20.752329 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.752311    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:20.752447 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:20.752377    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls podName:cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:28.752359535 +0000 UTC m=+51.162669609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls") pod "dns-default-44lg7" (UID: "cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:22.569223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:22.569183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:22.569682 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:22.569395    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:24:22.569682 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:22.569419    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd48b8b66-dnpdq: secret "image-registry-tls" not found
Sep 08 18:24:22.569682 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:22.569530    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls podName:c4d7b263-636a-4c20-9c72-bb56f51615f4 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:26.56950638 +0000 UTC m=+48.979816459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls") pod "image-registry-cd48b8b66-dnpdq" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4") : secret "image-registry-tls" not found
Sep 08 18:24:22.669656 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:22.669619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:22.669656 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:22.669659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:22.669840 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:22.669787    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:24:22.669840 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:22.669787    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:22.669907 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:22.669856    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls podName:40baa83d-6f8c-41f2-b74a-832246efe5cb nodeName:}" failed. No retries permitted until 2026-09-08 18:24:26.669838758 +0000 UTC m=+49.080148824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f6b9v" (UID: "40baa83d-6f8c-41f2-b74a-832246efe5cb") : secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:22.669907 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:22.669872    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert podName:4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc nodeName:}" failed. No retries permitted until 2026-09-08 18:24:26.669864481 +0000 UTC m=+49.080174540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-djllr" (UID: "4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc") : secret "networking-console-plugin-cert" not found
Sep 08 18:24:24.320004 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.319983    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wnh7g"]
Sep 08 18:24:24.335642 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.335608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.337069 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.337047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wnh7g"]
Sep 08 18:24:24.339983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.338972    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 18:24:24.339983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.339196    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-srf5g\""
Sep 08 18:24:24.339983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.339355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 18:24:24.339983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.339505    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 18:24:24.339983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.339631    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 18:24:24.348007 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.347981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr" event={"ID":"07dc1c3a-0077-48b4-a8f7-c6700e878e85","Type":"ContainerStarted","Data":"715b2e87485ec31e32ad8d9a4cd7f274c4c1905ecef26decda93b4d7d3eaa2d7"}
Sep 08 18:24:24.349721 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.349666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f" event={"ID":"1bf50b32-0985-4f48-87a0-ef0b751950db","Type":"ContainerStarted","Data":"6212f8bcfc3a730d5a7fa23f9db3e99bfcbf6de5316fe4f90d715de0e4197cac"}
Sep 08 18:24:24.372785 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.372744    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-s75dr" podStartSLOduration=1.917140412 podStartE2EDuration="6.372726161s" totalImagesPullingTime="4.455585749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:24.372159113 +0000 UTC m=+46.782469195" watchObservedRunningTime="2026-09-08 18:24:24.372726161 +0000 UTC m=+46.783036242"
Sep 08 18:24:24.483851 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.483749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6v8x\" (UniqueName: \"kubernetes.io/projected/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-api-access-b6v8x\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.483851 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.483811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.484067 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.483971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13e711ca-cb6f-4330-b5f8-4877403a29f9-crio-socket\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.484067 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.484034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.484067 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.484061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13e711ca-cb6f-4330-b5f8-4877403a29f9-data-volume\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.585400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.585477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13e711ca-cb6f-4330-b5f8-4877403a29f9-data-volume\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.585562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6v8x\" (UniqueName: \"kubernetes.io/projected/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-api-access-b6v8x\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:24.585578    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.585607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:24.585656    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:25.08563556 +0000 UTC m=+47.495945622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:24.585684    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:24.585755    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:25.085740016 +0000 UTC m=+47.496050089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:24.585768 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.585772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13e711ca-cb6f-4330-b5f8-4877403a29f9-crio-socket\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.586369 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.585865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13e711ca-cb6f-4330-b5f8-4877403a29f9-data-volume\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.586369 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.585963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13e711ca-cb6f-4330-b5f8-4877403a29f9-crio-socket\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:24.595632 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:24.595604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6v8x\" (UniqueName: \"kubernetes.io/projected/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-api-access-b6v8x\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:25.091373 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:25.091271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:25.091549 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:25.091398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:25.091549 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:25.091483    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:25.091549 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:25.091506    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:25.091672 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:25.091566    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:26.091547245 +0000 UTC m=+48.501857310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:25.091672 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:25.091583    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:26.091574444 +0000 UTC m=+48.501884506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:25.355015 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:25.354540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg" event={"ID":"d10402b8-2af5-478a-9663-b9efdaf54529","Type":"ContainerStarted","Data":"6fadf6bb757d3c8bfc26fef1d4dd98c3e0d3251111ebab68348d4494e57158d8"}
Sep 08 18:24:25.356261 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:25.356215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-ckf7z" event={"ID":"d79eeaa4-08bf-4e29-acfd-cf186424e109","Type":"ContainerStarted","Data":"f103002df48d2d78ca1d0320cbbe45080e47e083732e5924564ed686de865e60"}
Sep 08 18:24:25.371855 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:25.371802    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg" podStartSLOduration=2.6066627220000003 podStartE2EDuration="7.371785326s" totalImagesPullingTime="4.765122604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:25.371095319 +0000 UTC m=+47.781405401" watchObservedRunningTime="2026-09-08 18:24:25.371785326 +0000 UTC m=+47.782095411"
Sep 08 18:24:25.389343 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:25.389290    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-ckf7z" podStartSLOduration=2.6472328210000002 podStartE2EDuration="7.38927357s" totalImagesPullingTime="4.742040749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:25.385613746 +0000 UTC m=+47.795923830" watchObservedRunningTime="2026-09-08 18:24:25.38927357 +0000 UTC m=+47.799583651"
Sep 08 18:24:25.404100 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:25.404041    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f" podStartSLOduration=2.78758574 podStartE2EDuration="7.404026228s" totalImagesPullingTime="4.616440488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:25.402997568 +0000 UTC m=+47.813307646" watchObservedRunningTime="2026-09-08 18:24:25.404026228 +0000 UTC m=+47.814336310"
Sep 08 18:24:26.048970 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:26.048928    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lgmvk_31debb12-8b51-43d8-8c51-4bba3cd84bcd/dns-node-resolver/0.log"
Sep 08 18:24:26.101337 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:26.101296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:26.101536 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:26.101403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:26.101536 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.101455    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:26.101624 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.101543    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:28.101521627 +0000 UTC m=+50.511831691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:26.101624 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.101550    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:26.101624 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.101601    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:28.101584256 +0000 UTC m=+50.511894338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:26.606605 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:26.606568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:26.607024 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.606687    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:24:26.607024 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.606699    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd48b8b66-dnpdq: secret "image-registry-tls" not found
Sep 08 18:24:26.607024 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.606748    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls podName:c4d7b263-636a-4c20-9c72-bb56f51615f4 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:34.606735198 +0000 UTC m=+57.017045271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls") pod "image-registry-cd48b8b66-dnpdq" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4") : secret "image-registry-tls" not found
Sep 08 18:24:26.707314 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:26.707281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:26.707314 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:26.707319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:26.707556 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.707442    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:24:26.707556 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.707444    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:26.707556 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.707495    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert podName:4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc nodeName:}" failed. No retries permitted until 2026-09-08 18:24:34.707481509 +0000 UTC m=+57.117791592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-djllr" (UID: "4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc") : secret "networking-console-plugin-cert" not found
Sep 08 18:24:26.707556 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:26.707524    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls podName:40baa83d-6f8c-41f2-b74a-832246efe5cb nodeName:}" failed. No retries permitted until 2026-09-08 18:24:34.707517559 +0000 UTC m=+57.117827617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f6b9v" (UID: "40baa83d-6f8c-41f2-b74a-832246efe5cb") : secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:27.250204 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:27.250176    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gb8zd_3337f71d-84e5-4047-b6e3-614c0143b182/node-ca/0.log"
Sep 08 18:24:27.849739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:27.849667    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7gf7v_97d26502-e9d6-4c71-92e3-46f65d798522/serve-healthcheck-canary/0.log"
Sep 08 18:24:28.120057 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:28.120026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:28.120245 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:28.120111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:28.120245 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:28.120149    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:28.120245 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:28.120212    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:32.120194896 +0000 UTC m=+54.530504969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:28.120423 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:28.120250    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:28.120423 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:28.120298    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:32.120286529 +0000 UTC m=+54.530596588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:28.826277 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:28.826238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:28.826486 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:28.826421    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:28.826533 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:28.826509    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls podName:cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:44.826493651 +0000 UTC m=+67.236803715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls") pod "dns-default-44lg7" (UID: "cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:32.152819 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:32.152777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:32.153256 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:32.152876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:32.153256 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:32.152911    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:32.153256 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:32.152974    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:32.153256 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:32.152983    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:40.152966896 +0000 UTC m=+62.563276995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:32.153256 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:32.153027    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:40.153013513 +0000 UTC m=+62.563323596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:34.675748 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:34.675704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls\") pod \"image-registry-cd48b8b66-dnpdq\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") " pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:34.676205 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:34.675839    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 18:24:34.676205 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:34.675851    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd48b8b66-dnpdq: secret "image-registry-tls" not found
Sep 08 18:24:34.676205 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:34.675909    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls podName:c4d7b263-636a-4c20-9c72-bb56f51615f4 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:50.675894477 +0000 UTC m=+73.086204536 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls") pod "image-registry-cd48b8b66-dnpdq" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4") : secret "image-registry-tls" not found
Sep 08 18:24:34.776784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:34.776747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:34.776865 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:34.776793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:34.776929 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:34.776911    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:34.776984 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:34.776967    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 18:24:34.777018 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:34.776971    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls podName:40baa83d-6f8c-41f2-b74a-832246efe5cb nodeName:}" failed. No retries permitted until 2026-09-08 18:24:50.776958284 +0000 UTC m=+73.187268342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f6b9v" (UID: "40baa83d-6f8c-41f2-b74a-832246efe5cb") : secret "cluster-monitoring-operator-tls" not found
Sep 08 18:24:34.777057 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:34.777028    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert podName:4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc nodeName:}" failed. No retries permitted until 2026-09-08 18:24:50.777016408 +0000 UTC m=+73.187326467 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-djllr" (UID: "4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc") : secret "networking-console-plugin-cert" not found
Sep 08 18:24:38.299617 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:38.299588    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvdhn"
Sep 08 18:24:40.219718 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:40.219646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:40.220134 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:40.219788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:40.220134 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:40.219822    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:40.220134 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:40.219903    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls podName:13e711ca-cb6f-4330-b5f8-4877403a29f9 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:56.21988156 +0000 UTC m=+78.630191619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnh7g" (UID: "13e711ca-cb6f-4330-b5f8-4877403a29f9") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:40.220366 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:40.220346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e711ca-cb6f-4330-b5f8-4877403a29f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:43.848985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:43.848951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:43.851521 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:43.851501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2p9\" (UniqueName: \"kubernetes.io/projected/24ca9f64-7590-47f3-8925-43b6f28056e2-kube-api-access-2t2p9\") pod \"network-check-target-tgnbz\" (UID: \"24ca9f64-7590-47f3-8925-43b6f28056e2\") " pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:43.949996 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:43.949955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:43.950170 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:43.950027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:43.953032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:43.953015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 18:24:43.953081 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:43.953015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 18:24:43.962706 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:43.962678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c98280f8-b5ea-4ad5-b872-7c3d9f9379dd-original-pull-secret\") pod \"global-pull-secret-syncer-m824h\" (UID: \"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd\") " pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:43.962828 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:43.962734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42d754f9-b01a-4e28-822e-79349dec9556-metrics-certs\") pod \"network-metrics-daemon-ndrsb\" (UID: \"42d754f9-b01a-4e28-822e-79349dec9556\") " pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:44.028706 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.028672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jmlkr\""
Sep 08 18:24:44.034084 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.034064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frqcg\""
Sep 08 18:24:44.037091 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.037077    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:44.037169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.037105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m824h"
Sep 08 18:24:44.042268 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.042248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndrsb"
Sep 08 18:24:44.316783 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.316733    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgnbz"]
Sep 08 18:24:44.321322 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:44.321299    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ca9f64_7590_47f3_8925_43b6f28056e2.slice/crio-f1f20a28c114a5d0e89ef4309b8d96af9016364d2dcbcafdbcc0de78fbab211f WatchSource:0}: Error finding container f1f20a28c114a5d0e89ef4309b8d96af9016364d2dcbcafdbcc0de78fbab211f: Status 404 returned error can't find the container with id f1f20a28c114a5d0e89ef4309b8d96af9016364d2dcbcafdbcc0de78fbab211f
Sep 08 18:24:44.327409 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.327379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m824h"]
Sep 08 18:24:44.331996 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.331976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ndrsb"]
Sep 08 18:24:44.333816 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:44.333784    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98280f8_b5ea_4ad5_b872_7c3d9f9379dd.slice/crio-2111d037a426c306621f519310bd87e33c5717c801fe03ec846ba6f11f77fc4c WatchSource:0}: Error finding container 2111d037a426c306621f519310bd87e33c5717c801fe03ec846ba6f11f77fc4c: Status 404 returned error can't find the container with id 2111d037a426c306621f519310bd87e33c5717c801fe03ec846ba6f11f77fc4c
Sep 08 18:24:44.335001 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:44.334981    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d754f9_b01a_4e28_822e_79349dec9556.slice/crio-a7fe53136fe243f755254091d79bae10c36f523c7da50ab0ec89f0763a926077 WatchSource:0}: Error finding container a7fe53136fe243f755254091d79bae10c36f523c7da50ab0ec89f0763a926077: Status 404 returned error can't find the container with id a7fe53136fe243f755254091d79bae10c36f523c7da50ab0ec89f0763a926077
Sep 08 18:24:44.407223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.407175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndrsb" event={"ID":"42d754f9-b01a-4e28-822e-79349dec9556","Type":"ContainerStarted","Data":"a7fe53136fe243f755254091d79bae10c36f523c7da50ab0ec89f0763a926077"}
Sep 08 18:24:44.408465 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.408418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m824h" event={"ID":"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd","Type":"ContainerStarted","Data":"2111d037a426c306621f519310bd87e33c5717c801fe03ec846ba6f11f77fc4c"}
Sep 08 18:24:44.409521 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.409498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgnbz" event={"ID":"24ca9f64-7590-47f3-8925-43b6f28056e2","Type":"ContainerStarted","Data":"f1f20a28c114a5d0e89ef4309b8d96af9016364d2dcbcafdbcc0de78fbab211f"}
Sep 08 18:24:44.858395 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.858349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:44.861060 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:44.861027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5-metrics-tls\") pod \"dns-default-44lg7\" (UID: \"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5\") " pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:45.055058 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:45.055022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vbqmk\""
Sep 08 18:24:45.063391 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:45.063366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:45.344049 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:45.343964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-44lg7"]
Sep 08 18:24:45.416527 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:45.416421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgnbz" event={"ID":"24ca9f64-7590-47f3-8925-43b6f28056e2","Type":"ContainerStarted","Data":"2526a91b3e36f5b867a010fe2b74a6e54c80a08c36594b2f139e6afef6f9b367"}
Sep 08 18:24:45.416685 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:45.416540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:24:45.434622 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:45.434512    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tgnbz" podStartSLOduration=66.434493192 podStartE2EDuration="1m6.434493192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:45.431733312 +0000 UTC m=+67.842043392" watchObservedRunningTime="2026-09-08 18:24:45.434493192 +0000 UTC m=+67.844803274"
Sep 08 18:24:45.618207 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:45.618170    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8f8cfc_9c85_4e3d_ad27_7e45ff816ad5.slice/crio-56550a7ac9dcfd4cd2df4da36002127891064f86b1d9e737d7d3ac9abde77b39 WatchSource:0}: Error finding container 56550a7ac9dcfd4cd2df4da36002127891064f86b1d9e737d7d3ac9abde77b39: Status 404 returned error can't find the container with id 56550a7ac9dcfd4cd2df4da36002127891064f86b1d9e737d7d3ac9abde77b39
Sep 08 18:24:46.422294 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:46.422257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndrsb" event={"ID":"42d754f9-b01a-4e28-822e-79349dec9556","Type":"ContainerStarted","Data":"2cfd00b0a7e8717521d29977aef76240af77186a6f9864cd3a74b2e312894d8a"}
Sep 08 18:24:46.422782 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:46.422325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndrsb" event={"ID":"42d754f9-b01a-4e28-822e-79349dec9556","Type":"ContainerStarted","Data":"2ffebbcc185d66feeabdf3dbf631af217501a7de9b1ce857acce599550a55ae5"}
Sep 08 18:24:46.424082 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:46.424055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44lg7" event={"ID":"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5","Type":"ContainerStarted","Data":"56550a7ac9dcfd4cd2df4da36002127891064f86b1d9e737d7d3ac9abde77b39"}
Sep 08 18:24:47.829098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.829032    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ndrsb" podStartSLOduration=67.442787351 podStartE2EDuration="1m8.829013783s" totalImagesPullingTime="1.386226432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:46.437883154 +0000 UTC m=+68.848193238" watchObservedRunningTime="2026-09-08 18:24:47.829013783 +0000 UTC m=+70.239323895"
Sep 08 18:24:47.830018 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.829987    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"]
Sep 08 18:24:47.833159 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.833131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.838490 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.838465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 18:24:47.838896 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.838876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 18:24:47.839010 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.838876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 18:24:47.840845 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.840825    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 18:24:47.846035 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.846014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"]
Sep 08 18:24:47.886769 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.886734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983af01d-76c7-456a-ba49-6e212df9ad88-tmp\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.886769 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.886769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/983af01d-76c7-456a-ba49-6e212df9ad88-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.887023 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.886898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75cmt\" (UniqueName: \"kubernetes.io/projected/983af01d-76c7-456a-ba49-6e212df9ad88-kube-api-access-75cmt\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.901165 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.901130    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cd48b8b66-dnpdq"]
Sep 08 18:24:47.901461 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:47.901412    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq" podUID="c4d7b263-636a-4c20-9c72-bb56f51615f4"
Sep 08 18:24:47.946890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.946854    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cb99ddd65-v9zg2"]
Sep 08 18:24:47.950137 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.950108    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:47.964260 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.964238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cb99ddd65-v9zg2"]
Sep 08 18:24:47.987286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.987255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983af01d-76c7-456a-ba49-6e212df9ad88-tmp\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.987448 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.987296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/983af01d-76c7-456a-ba49-6e212df9ad88-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.987448 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.987353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75cmt\" (UniqueName: \"kubernetes.io/projected/983af01d-76c7-456a-ba49-6e212df9ad88-kube-api-access-75cmt\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.987897 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.987870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983af01d-76c7-456a-ba49-6e212df9ad88-tmp\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.990441 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.990405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/983af01d-76c7-456a-ba49-6e212df9ad88-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:47.997174 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:47.997148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75cmt\" (UniqueName: \"kubernetes.io/projected/983af01d-76c7-456a-ba49-6e212df9ad88-kube-api-access-75cmt\") pod \"klusterlet-addon-workmgr-56c5cf5cb5-rbcrs\" (UID: \"983af01d-76c7-456a-ba49-6e212df9ad88\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:48.088628 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.088535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-registry-tls\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.088628 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.088593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7751a12e-8b39-4591-9bed-b91830663aa2-ca-trust-extracted\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.088830 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.088669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-bound-sa-token\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.088830 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.088706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7751a12e-8b39-4591-9bed-b91830663aa2-registry-certificates\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.088830 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.088814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbjv\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-kube-api-access-trbjv\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.088948 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.088843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7751a12e-8b39-4591-9bed-b91830663aa2-image-registry-private-configuration\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.088948 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.088860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7751a12e-8b39-4591-9bed-b91830663aa2-trusted-ca\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.088948 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.088888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7751a12e-8b39-4591-9bed-b91830663aa2-installation-pull-secrets\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.143947 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.143918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:48.189223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7751a12e-8b39-4591-9bed-b91830663aa2-installation-pull-secrets\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.189390 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-registry-tls\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.189390 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7751a12e-8b39-4591-9bed-b91830663aa2-ca-trust-extracted\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.189390 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-bound-sa-token\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.189390 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7751a12e-8b39-4591-9bed-b91830663aa2-registry-certificates\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.189630 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbjv\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-kube-api-access-trbjv\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.189630 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7751a12e-8b39-4591-9bed-b91830663aa2-image-registry-private-configuration\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.189630 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7751a12e-8b39-4591-9bed-b91830663aa2-trusted-ca\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.189873 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.189852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7751a12e-8b39-4591-9bed-b91830663aa2-ca-trust-extracted\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.190355 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.190323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7751a12e-8b39-4591-9bed-b91830663aa2-trusted-ca\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.190498 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.190466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7751a12e-8b39-4591-9bed-b91830663aa2-registry-certificates\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.192196 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.192178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7751a12e-8b39-4591-9bed-b91830663aa2-installation-pull-secrets\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.192533 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.192511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7751a12e-8b39-4591-9bed-b91830663aa2-image-registry-private-configuration\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.194277 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.194253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-registry-tls\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.197946 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.197924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-bound-sa-token\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.199706 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.199685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbjv\" (UniqueName: \"kubernetes.io/projected/7751a12e-8b39-4591-9bed-b91830663aa2-kube-api-access-trbjv\") pod \"image-registry-5cb99ddd65-v9zg2\" (UID: \"7751a12e-8b39-4591-9bed-b91830663aa2\") " pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.264628 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.264598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cm2c5\""
Sep 08 18:24:48.271960 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.271940    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:48.431511 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.431479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:48.431673 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.431483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44lg7" event={"ID":"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5","Type":"ContainerStarted","Data":"8066461c31657109361469280e8bf70296f732a490ac7224d217536f5533f2b2"}
Sep 08 18:24:48.438991 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.438971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:48.491169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491140    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-image-registry-private-configuration\") pod \"c4d7b263-636a-4c20-9c72-bb56f51615f4\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") "
Sep 08 18:24:48.491325 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-installation-pull-secrets\") pod \"c4d7b263-636a-4c20-9c72-bb56f51615f4\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") "
Sep 08 18:24:48.491325 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hwxm\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-kube-api-access-7hwxm\") pod \"c4d7b263-636a-4c20-9c72-bb56f51615f4\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") "
Sep 08 18:24:48.491325 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491258    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d7b263-636a-4c20-9c72-bb56f51615f4-ca-trust-extracted\") pod \"c4d7b263-636a-4c20-9c72-bb56f51615f4\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") "
Sep 08 18:24:48.491325 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-bound-sa-token\") pod \"c4d7b263-636a-4c20-9c72-bb56f51615f4\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") "
Sep 08 18:24:48.491610 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491581    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d7b263-636a-4c20-9c72-bb56f51615f4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c4d7b263-636a-4c20-9c72-bb56f51615f4" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:24:48.491681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-certificates\") pod \"c4d7b263-636a-4c20-9c72-bb56f51615f4\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") "
Sep 08 18:24:48.491750 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-trusted-ca\") pod \"c4d7b263-636a-4c20-9c72-bb56f51615f4\" (UID: \"c4d7b263-636a-4c20-9c72-bb56f51615f4\") "
Sep 08 18:24:48.491992 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c4d7b263-636a-4c20-9c72-bb56f51615f4" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:24:48.491992 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.491981    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4d7b263-636a-4c20-9c72-bb56f51615f4-ca-trust-extracted\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:24:48.492376 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.492348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c4d7b263-636a-4c20-9c72-bb56f51615f4" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:24:48.494071 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.494040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c4d7b263-636a-4c20-9c72-bb56f51615f4" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:24:48.494180 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.494043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c4d7b263-636a-4c20-9c72-bb56f51615f4" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:24:48.494180 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.494108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-kube-api-access-7hwxm" (OuterVolumeSpecName: "kube-api-access-7hwxm") pod "c4d7b263-636a-4c20-9c72-bb56f51615f4" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4"). InnerVolumeSpecName "kube-api-access-7hwxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:24:48.494522 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.494492    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c4d7b263-636a-4c20-9c72-bb56f51615f4" (UID: "c4d7b263-636a-4c20-9c72-bb56f51615f4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:24:48.592935 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.592899    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-image-registry-private-configuration\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:24:48.592935 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.592931    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4d7b263-636a-4c20-9c72-bb56f51615f4-installation-pull-secrets\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:24:48.593166 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.592948    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwxm\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-kube-api-access-7hwxm\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:24:48.593166 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.592957    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-bound-sa-token\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:24:48.593166 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.592966    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-certificates\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:24:48.593166 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:48.592974    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4d7b263-636a-4c20-9c72-bb56f51615f4-trusted-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:24:49.127928 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.127904    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"]
Sep 08 18:24:49.134846 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:49.134813    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751a12e_8b39_4591_9bed_b91830663aa2.slice/crio-283766f5fe000cf090ccaf8460e61d40a4d6a2825bd6cf7adbd5e47d4868fc7c WatchSource:0}: Error finding container 283766f5fe000cf090ccaf8460e61d40a4d6a2825bd6cf7adbd5e47d4868fc7c: Status 404 returned error can't find the container with id 283766f5fe000cf090ccaf8460e61d40a4d6a2825bd6cf7adbd5e47d4868fc7c
Sep 08 18:24:49.135078 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.135054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cb99ddd65-v9zg2"]
Sep 08 18:24:49.436918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.436829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44lg7" event={"ID":"cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5","Type":"ContainerStarted","Data":"ce7faf3cb37c6e4e9a3f9e980701826ad6e47b8aa9fa6cd2dd5d368f3bd653ce"}
Sep 08 18:24:49.437063 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.437037    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:49.440365 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.440326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m824h" event={"ID":"c98280f8-b5ea-4ad5-b872-7c3d9f9379dd","Type":"ContainerStarted","Data":"d3ec4c72eec760a8d9d3d025ccb706585d9e82364b0d0cd3a228d33028fd5d0a"}
Sep 08 18:24:49.442322 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.442292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2" event={"ID":"7751a12e-8b39-4591-9bed-b91830663aa2","Type":"ContainerStarted","Data":"8d1a2f7b91a1a6cf5451533895d36794765d5e79ce43e39bc72e6ff253ddfa51"}
Sep 08 18:24:49.442457 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.442342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2" event={"ID":"7751a12e-8b39-4591-9bed-b91830663aa2","Type":"ContainerStarted","Data":"283766f5fe000cf090ccaf8460e61d40a4d6a2825bd6cf7adbd5e47d4868fc7c"}
Sep 08 18:24:49.442457 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.442375    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:24:49.443467 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.443447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs" event={"ID":"983af01d-76c7-456a-ba49-6e212df9ad88","Type":"ContainerStarted","Data":"7e290dc18600122498fbe5f5ae00d12267c6b93234824e764a96909dde27f1f5"}
Sep 08 18:24:49.443545 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.443455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd48b8b66-dnpdq"
Sep 08 18:24:49.457063 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.457028    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-44lg7" podStartSLOduration=35.723919046 podStartE2EDuration="37.4570156s" totalImagesPullingTime="1.733096554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:49.455543414 +0000 UTC m=+71.865853496" watchObservedRunningTime="2026-09-08 18:24:49.4570156 +0000 UTC m=+71.867325680"
Sep 08 18:24:49.500776 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.500746    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cd48b8b66-dnpdq"]
Sep 08 18:24:49.502843 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.502819    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cd48b8b66-dnpdq"]
Sep 08 18:24:49.516230 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.516192    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m824h" podStartSLOduration=65.929459091 podStartE2EDuration="1m10.516178663s" totalImagesPullingTime="4.586719572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:49.515417514 +0000 UTC m=+71.925727594" watchObservedRunningTime="2026-09-08 18:24:49.516178663 +0000 UTC m=+71.926488738"
Sep 08 18:24:49.534756 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.534710    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2" podStartSLOduration=2.534696205 podStartE2EDuration="2.534696205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:49.533929526 +0000 UTC m=+71.944239608" watchObservedRunningTime="2026-09-08 18:24:49.534696205 +0000 UTC m=+71.945006286"
Sep 08 18:24:49.602672 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:49.602644    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4d7b263-636a-4c20-9c72-bb56f51615f4-registry-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:24:50.120812 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:50.120756    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d7b263-636a-4c20-9c72-bb56f51615f4" path="/var/lib/kubelet/pods/c4d7b263-636a-4c20-9c72-bb56f51615f4/volumes"
Sep 08 18:24:50.813969 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:50.813931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:50.814459 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:50.813986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:50.817013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:50.816987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-djllr\" (UID: \"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:50.817115 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:50.817064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/40baa83d-6f8c-41f2-b74a-832246efe5cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f6b9v\" (UID: \"40baa83d-6f8c-41f2-b74a-832246efe5cb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:51.018347 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:51.018315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8nhwz\""
Sep 08 18:24:51.026508 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:51.026479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"
Sep 08 18:24:51.063578 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:51.063546    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ttsc8\""
Sep 08 18:24:51.071206 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:51.071139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-djllr"
Sep 08 18:24:51.714459 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:51.714386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-djllr"]
Sep 08 18:24:51.724134 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:51.723969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v"]
Sep 08 18:24:52.924684 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:52.924655    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa09f9c_1d0e_427d_99fc_cc7c0aa637dc.slice/crio-711948cbf58f35b4e51d0857ad548e60815bd13f1280d21638ffec199c0fd8ba WatchSource:0}: Error finding container 711948cbf58f35b4e51d0857ad548e60815bd13f1280d21638ffec199c0fd8ba: Status 404 returned error can't find the container with id 711948cbf58f35b4e51d0857ad548e60815bd13f1280d21638ffec199c0fd8ba
Sep 08 18:24:52.925136 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:52.925112    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40baa83d_6f8c_41f2_b74a_832246efe5cb.slice/crio-03c23472700788a1ee86af28b4798f866d573cf9bd0f5dde347c4acf9e1fd767 WatchSource:0}: Error finding container 03c23472700788a1ee86af28b4798f866d573cf9bd0f5dde347c4acf9e1fd767: Status 404 returned error can't find the container with id 03c23472700788a1ee86af28b4798f866d573cf9bd0f5dde347c4acf9e1fd767
Sep 08 18:24:53.461055 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:53.461014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-djllr" event={"ID":"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc","Type":"ContainerStarted","Data":"711948cbf58f35b4e51d0857ad548e60815bd13f1280d21638ffec199c0fd8ba"}
Sep 08 18:24:53.462309 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:53.462286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v" event={"ID":"40baa83d-6f8c-41f2-b74a-832246efe5cb","Type":"ContainerStarted","Data":"03c23472700788a1ee86af28b4798f866d573cf9bd0f5dde347c4acf9e1fd767"}
Sep 08 18:24:54.468269 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:54.468227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs" event={"ID":"983af01d-76c7-456a-ba49-6e212df9ad88","Type":"ContainerStarted","Data":"d2cc43b389a1ed50e05fcd9920132a65a7b38cd7552252edd0552d1dba30a0a2"}
Sep 08 18:24:54.468738 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:54.468445    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:54.470242 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:54.470221    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs"
Sep 08 18:24:54.485196 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:54.485150    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c5cf5cb5-rbcrs" podStartSLOduration=3.247098397 podStartE2EDuration="7.485135971s" totalImagesPullingTime="4.238037574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:54.483819923 +0000 UTC m=+76.894130004" watchObservedRunningTime="2026-09-08 18:24:54.485135971 +0000 UTC m=+76.895446051"
Sep 08 18:24:55.473004 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:55.472968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-djllr" event={"ID":"4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc","Type":"ContainerStarted","Data":"b1fefe2e16abe3a823ea4b8daed45c800645da66d4563409dff15a066797a65e"}
Sep 08 18:24:55.488058 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:55.487985    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-djllr" podStartSLOduration=35.178810175 podStartE2EDuration="37.487969952s" totalImagesPullingTime="2.309159777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:55.487701436 +0000 UTC m=+77.898011528" watchObservedRunningTime="2026-09-08 18:24:55.487969952 +0000 UTC m=+77.898280032"
Sep 08 18:24:56.043862 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.042102    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"]
Sep 08 18:24:56.045697 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.045668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"
Sep 08 18:24:56.048701 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.048681    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 18:24:56.048701 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.048689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tdqjv\""
Sep 08 18:24:56.052567 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.052548    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"]
Sep 08 18:24:56.162861 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.162832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf3e1995-8c3e-436c-83d3-ea2f38fcd01e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wbgc9\" (UID: \"cf3e1995-8c3e-436c-83d3-ea2f38fcd01e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"
Sep 08 18:24:56.263678 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.263639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:56.263678 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.263682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf3e1995-8c3e-436c-83d3-ea2f38fcd01e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wbgc9\" (UID: \"cf3e1995-8c3e-436c-83d3-ea2f38fcd01e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"
Sep 08 18:24:56.263909 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:56.263816    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 18:24:56.263909 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:24:56.263896    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3e1995-8c3e-436c-83d3-ea2f38fcd01e-tls-certificates podName:cf3e1995-8c3e-436c-83d3-ea2f38fcd01e nodeName:}" failed. No retries permitted until 2026-09-08 18:24:56.763875513 +0000 UTC m=+79.174185589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/cf3e1995-8c3e-436c-83d3-ea2f38fcd01e-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-wbgc9" (UID: "cf3e1995-8c3e-436c-83d3-ea2f38fcd01e") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 18:24:56.266136 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.266107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e711ca-cb6f-4330-b5f8-4877403a29f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnh7g\" (UID: \"13e711ca-cb6f-4330-b5f8-4877403a29f9\") " pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:56.464808 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.464778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-srf5g\""
Sep 08 18:24:56.472082 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.472061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wnh7g"
Sep 08 18:24:56.477947 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.477917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v" event={"ID":"40baa83d-6f8c-41f2-b74a-832246efe5cb","Type":"ContainerStarted","Data":"06b38bdbeafac7df931e7171a699ee276e0ac6858ad2ff7888cb85cef58b9b39"}
Sep 08 18:24:56.497562 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.497498    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f6b9v" podStartSLOduration=35.930392817 podStartE2EDuration="38.497474962s" totalImagesPullingTime="2.567082145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:56.495902606 +0000 UTC m=+78.906212699" watchObservedRunningTime="2026-09-08 18:24:56.497474962 +0000 UTC m=+78.907785044"
Sep 08 18:24:56.749792 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.749598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wnh7g"]
Sep 08 18:24:56.752402 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:56.752377    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e711ca_cb6f_4330_b5f8_4877403a29f9.slice/crio-88078f60f07f1b5fad1feb4a5b60c57659cbfa0e7f9cab2527f1e54f099956f3 WatchSource:0}: Error finding container 88078f60f07f1b5fad1feb4a5b60c57659cbfa0e7f9cab2527f1e54f099956f3: Status 404 returned error can't find the container with id 88078f60f07f1b5fad1feb4a5b60c57659cbfa0e7f9cab2527f1e54f099956f3
Sep 08 18:24:56.768043 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.768023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf3e1995-8c3e-436c-83d3-ea2f38fcd01e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wbgc9\" (UID: \"cf3e1995-8c3e-436c-83d3-ea2f38fcd01e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"
Sep 08 18:24:56.770899 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.770874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf3e1995-8c3e-436c-83d3-ea2f38fcd01e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wbgc9\" (UID: \"cf3e1995-8c3e-436c-83d3-ea2f38fcd01e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"
Sep 08 18:24:56.955186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:56.955140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"
Sep 08 18:24:57.224858 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.224833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"]
Sep 08 18:24:57.227090 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:57.227064    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3e1995_8c3e_436c_83d3_ea2f38fcd01e.slice/crio-6c86e9eda6a068cda1818733a872f3eed6a00efbe119c15fa0c475714240ca8f WatchSource:0}: Error finding container 6c86e9eda6a068cda1818733a872f3eed6a00efbe119c15fa0c475714240ca8f: Status 404 returned error can't find the container with id 6c86e9eda6a068cda1818733a872f3eed6a00efbe119c15fa0c475714240ca8f
Sep 08 18:24:57.482307 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.482275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9" event={"ID":"cf3e1995-8c3e-436c-83d3-ea2f38fcd01e","Type":"ContainerStarted","Data":"6c86e9eda6a068cda1818733a872f3eed6a00efbe119c15fa0c475714240ca8f"}
Sep 08 18:24:57.483719 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.483697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnh7g" event={"ID":"13e711ca-cb6f-4330-b5f8-4877403a29f9","Type":"ContainerStarted","Data":"dbb29d364bd2f9729ea9c42f3f38db02f53ba8e59595a4dc75e57e0135afbcb0"}
Sep 08 18:24:57.483824 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.483742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnh7g" event={"ID":"13e711ca-cb6f-4330-b5f8-4877403a29f9","Type":"ContainerStarted","Data":"88078f60f07f1b5fad1feb4a5b60c57659cbfa0e7f9cab2527f1e54f099956f3"}
Sep 08 18:24:57.637762 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.637728    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d6df45f77-b82nb"]
Sep 08 18:24:57.640753 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.640735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.643783 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.643764    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 18:24:57.645458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.645306    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 18:24:57.645458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.645341    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 18:24:57.645458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.645366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 18:24:57.645458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.645383    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 18:24:57.645458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.645373    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 18:24:57.645747 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.645627    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 18:24:57.645747 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.645703    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vdqzb\""
Sep 08 18:24:57.649991 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.649971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d6df45f77-b82nb"]
Sep 08 18:24:57.777044 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.776957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-config\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.777044 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.777008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gmk\" (UniqueName: \"kubernetes.io/projected/b0fdf782-84b3-4d87-80b4-5a7647b16827-kube-api-access-f4gmk\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.777223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.777071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-service-ca\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.777223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.777092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-oauth-serving-cert\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.777223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.777149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-oauth-config\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.777223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.777200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-serving-cert\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.878469 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.878422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-service-ca\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.878469 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.878472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-oauth-serving-cert\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.878718 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.878494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-oauth-config\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.878718 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.878522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-serving-cert\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.878718 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.878558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-config\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.878718 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.878611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gmk\" (UniqueName: \"kubernetes.io/projected/b0fdf782-84b3-4d87-80b4-5a7647b16827-kube-api-access-f4gmk\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.879240 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.879191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-oauth-serving-cert\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.879346 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.879285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-service-ca\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.879346 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.879310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-config\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.881445 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.881409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-serving-cert\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.882148 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.882123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-oauth-config\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.887119 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.887099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gmk\" (UniqueName: \"kubernetes.io/projected/b0fdf782-84b3-4d87-80b4-5a7647b16827-kube-api-access-f4gmk\") pod \"console-d6df45f77-b82nb\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") " pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:57.950372 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:57.950323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:24:58.234210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:58.234176    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d6df45f77-b82nb"]
Sep 08 18:24:58.434056 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:24:58.434021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fdf782_84b3_4d87_80b4_5a7647b16827.slice/crio-072d8edc44ddd4c94a6150dd0e6755502763aca1038d7fabcb2d2f1d0b481514 WatchSource:0}: Error finding container 072d8edc44ddd4c94a6150dd0e6755502763aca1038d7fabcb2d2f1d0b481514: Status 404 returned error can't find the container with id 072d8edc44ddd4c94a6150dd0e6755502763aca1038d7fabcb2d2f1d0b481514
Sep 08 18:24:58.488573 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:58.488540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnh7g" event={"ID":"13e711ca-cb6f-4330-b5f8-4877403a29f9","Type":"ContainerStarted","Data":"1b3bdc4a15fcdf7eb630a7df22f86463d36a1f75c6c2f6b01e51d1049a6cf8ad"}
Sep 08 18:24:58.489809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:58.489786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6df45f77-b82nb" event={"ID":"b0fdf782-84b3-4d87-80b4-5a7647b16827","Type":"ContainerStarted","Data":"072d8edc44ddd4c94a6150dd0e6755502763aca1038d7fabcb2d2f1d0b481514"}
Sep 08 18:24:59.450719 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:59.450556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-44lg7"
Sep 08 18:24:59.496293 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:59.496257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9" event={"ID":"cf3e1995-8c3e-436c-83d3-ea2f38fcd01e","Type":"ContainerStarted","Data":"7220f31dbee2754cf0c44649ad63745f30fe7e7b2ab8cbe43524fe8577d66a06"}
Sep 08 18:24:59.496774 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:59.496422    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"
Sep 08 18:24:59.498776 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:59.498751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnh7g" event={"ID":"13e711ca-cb6f-4330-b5f8-4877403a29f9","Type":"ContainerStarted","Data":"db13a581a332158d1aa10dbe129dd742751361280a26593a78f565df6f7fbe5e"}
Sep 08 18:24:59.501556 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:59.501535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9"
Sep 08 18:24:59.513335 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:59.513292    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wbgc9" podStartSLOduration=2.264184094 podStartE2EDuration="3.513277249s" totalImagesPullingTime="1.249093155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:59.513055077 +0000 UTC m=+81.923365158" watchObservedRunningTime="2026-09-08 18:24:59.513277249 +0000 UTC m=+81.923587330"
Sep 08 18:24:59.532204 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:24:59.532155    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wnh7g" podStartSLOduration=33.247481505 podStartE2EDuration="35.532136503s" totalImagesPullingTime="2.284654998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:24:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:59.53071127 +0000 UTC m=+81.941021366" watchObservedRunningTime="2026-09-08 18:24:59.532136503 +0000 UTC m=+81.942446585"
Sep 08 18:25:02.511630 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:02.511589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6df45f77-b82nb" event={"ID":"b0fdf782-84b3-4d87-80b4-5a7647b16827","Type":"ContainerStarted","Data":"fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241"}
Sep 08 18:25:02.532645 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:02.532591    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d6df45f77-b82nb" podStartSLOduration=2.306835639 podStartE2EDuration="5.532577037s" totalImagesPullingTime="3.225741398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:02.532383983 +0000 UTC m=+84.942694064" watchObservedRunningTime="2026-09-08 18:25:02.532577037 +0000 UTC m=+84.942887119"
Sep 08 18:25:07.950496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:07.950446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:25:07.950496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:07.950510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:25:07.955332 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:07.955309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:25:08.536981 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:08.536953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:25:10.451827 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:10.451799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cb99ddd65-v9zg2"
Sep 08 18:25:11.519442 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.519401    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lmh5d"]
Sep 08 18:25:11.524142 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.524118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"]
Sep 08 18:25:11.524297 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.524264    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.527204 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.527183    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 18:25:11.527314 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.527278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4lkk\""
Sep 08 18:25:11.527800 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.527787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.528383 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.528365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 18:25:11.528383 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.528371    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 18:25:11.528558 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.528413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 18:25:11.531939 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.531920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 18:25:11.532307 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.532292    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-g8bfk\""
Sep 08 18:25:11.532383 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.532326    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 18:25:11.532603 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.532587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 18:25:11.543824 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.543801    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"]
Sep 08 18:25:11.687352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42917c73-bc39-43cc-8538-d9301838cc0b-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.687352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-sys\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.687583 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-accelerators-collector-config\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.687583 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjmz\" (UniqueName: \"kubernetes.io/projected/c90e83c6-3962-4dd5-97fd-050165a9c846-kube-api-access-2jjmz\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.687583 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42917c73-bc39-43cc-8538-d9301838cc0b-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.687583 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-root\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.687583 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.687737 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.687737 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-tls\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.687737 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-wtmp\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.687737 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c90e83c6-3962-4dd5-97fd-050165a9c846-metrics-client-ca\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.687737 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.687737 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czd87\" (UniqueName: \"kubernetes.io/projected/42917c73-bc39-43cc-8538-d9301838cc0b-kube-api-access-czd87\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.687912 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.687912 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.687809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-textfile\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.788862 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.788795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.788862 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.788841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-textfile\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789020 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.788881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42917c73-bc39-43cc-8538-d9301838cc0b-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.789020 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.788911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-sys\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789020 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.788938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-accelerators-collector-config\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789020 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.788972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjmz\" (UniqueName: \"kubernetes.io/projected/c90e83c6-3962-4dd5-97fd-050165a9c846-kube-api-access-2jjmz\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789020 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.788993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42917c73-bc39-43cc-8538-d9301838cc0b-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.789020 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-root\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-sys\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-root\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-textfile\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.789356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-tls\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-wtmp\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c90e83c6-3962-4dd5-97fd-050165a9c846-metrics-client-ca\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.789681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czd87\" (UniqueName: \"kubernetes.io/projected/42917c73-bc39-43cc-8538-d9301838cc0b-kube-api-access-czd87\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.789681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-wtmp\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-accelerators-collector-config\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.789681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.790034 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42917c73-bc39-43cc-8538-d9301838cc0b-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.790034 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.789819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42917c73-bc39-43cc-8538-d9301838cc0b-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.790140 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.790097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c90e83c6-3962-4dd5-97fd-050165a9c846-metrics-client-ca\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.791876 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.791856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.791980 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.791945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42917c73-bc39-43cc-8538-d9301838cc0b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.792244 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.792220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.792296 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.792289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c90e83c6-3962-4dd5-97fd-050165a9c846-node-exporter-tls\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.798025 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.797998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czd87\" (UniqueName: \"kubernetes.io/projected/42917c73-bc39-43cc-8538-d9301838cc0b-kube-api-access-czd87\") pod \"kube-state-metrics-74cb57df7-5b6rv\" (UID: \"42917c73-bc39-43cc-8538-d9301838cc0b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.798139 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.798066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjmz\" (UniqueName: \"kubernetes.io/projected/c90e83c6-3962-4dd5-97fd-050165a9c846-kube-api-access-2jjmz\") pod \"node-exporter-lmh5d\" (UID: \"c90e83c6-3962-4dd5-97fd-050165a9c846\") " pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.835214 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.835177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lmh5d"
Sep 08 18:25:11.839189 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:11.839175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"
Sep 08 18:25:11.849285 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:25:11.849252    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90e83c6_3962_4dd5_97fd_050165a9c846.slice/crio-f0ec3b0b19aeacde8459423c2941be39dbafdf006685d3858c9552b7ee4a483a WatchSource:0}: Error finding container f0ec3b0b19aeacde8459423c2941be39dbafdf006685d3858c9552b7ee4a483a: Status 404 returned error can't find the container with id f0ec3b0b19aeacde8459423c2941be39dbafdf006685d3858c9552b7ee4a483a
Sep 08 18:25:12.112086 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.112003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv"]
Sep 08 18:25:12.114865 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:25:12.114837    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42917c73_bc39_43cc_8538_d9301838cc0b.slice/crio-94276e994f9d46c40c8d96ca5467b6739a9b6f04994c7ef994b6aa631bbe16d1 WatchSource:0}: Error finding container 94276e994f9d46c40c8d96ca5467b6739a9b6f04994c7ef994b6aa631bbe16d1: Status 404 returned error can't find the container with id 94276e994f9d46c40c8d96ca5467b6739a9b6f04994c7ef994b6aa631bbe16d1
Sep 08 18:25:12.548368 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.548331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmh5d" event={"ID":"c90e83c6-3962-4dd5-97fd-050165a9c846","Type":"ContainerStarted","Data":"f0ec3b0b19aeacde8459423c2941be39dbafdf006685d3858c9552b7ee4a483a"}
Sep 08 18:25:12.549611 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.549581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv" event={"ID":"42917c73-bc39-43cc-8538-d9301838cc0b","Type":"ContainerStarted","Data":"94276e994f9d46c40c8d96ca5467b6739a9b6f04994c7ef994b6aa631bbe16d1"}
Sep 08 18:25:12.579656 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.579623    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:25:12.583208 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.583187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.588617 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.588585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 18:25:12.588735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.588719    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 18:25:12.588810 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.588803    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 18:25:12.589254 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.589234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 18:25:12.589325 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.589300    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 18:25:12.589827 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.589809    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 18:25:12.589914 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.589828    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 18:25:12.589914 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.589828    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 18:25:12.590029 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.589910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xxcn7\""
Sep 08 18:25:12.590546 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.590529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 18:25:12.604309 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.604286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:25:12.697232 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46w8\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-kube-api-access-m46w8\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697410 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697410 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697410 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-web-config\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697410 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697410 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697649 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697649 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697649 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697649 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697649 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-out\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697649 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.697649 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.697602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.798963 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.798876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46w8\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-kube-api-access-m46w8\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.798963 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.798931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799277 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799334 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-web-config\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799387 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799387 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799474 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799514 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799514 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799586 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799586 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-out\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799586 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.799697 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.799634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.800074 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.800005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.800643 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.800617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.805615 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.805111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.806769 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.806274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.807932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.807907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-out\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.808504 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.808483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.810601 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.810549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.812280 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.812254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.813590 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.813566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-web-config\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.821341 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.820647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.821341 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.821103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.821341 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.821266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.823240 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.823210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46w8\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-kube-api-access-m46w8\") pod \"alertmanager-main-0\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:12.894020 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:12.893792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:25:13.318322 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:13.318105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:25:13.556212 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:13.556175    2580 generic.go:356] "Generic (PLEG): container finished" podID="c90e83c6-3962-4dd5-97fd-050165a9c846" containerID="66f44f58d8d96c50edbccd3922fbcf7f7eb59ca62da8cf1e07ca44ce28f6da7e" exitCode=0
Sep 08 18:25:13.556661 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:13.556241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmh5d" event={"ID":"c90e83c6-3962-4dd5-97fd-050165a9c846","Type":"ContainerDied","Data":"66f44f58d8d96c50edbccd3922fbcf7f7eb59ca62da8cf1e07ca44ce28f6da7e"}
Sep 08 18:25:13.589333 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:25:13.589299    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb397ae8f_85d4_4a37_bbf9_01851739b1a2.slice/crio-b40845dc9495c4382e63ad9574a9d48e46c4e7d5efce9a156f389e3b47b078bd WatchSource:0}: Error finding container b40845dc9495c4382e63ad9574a9d48e46c4e7d5efce9a156f389e3b47b078bd: Status 404 returned error can't find the container with id b40845dc9495c4382e63ad9574a9d48e46c4e7d5efce9a156f389e3b47b078bd
Sep 08 18:25:14.562517 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:14.562475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmh5d" event={"ID":"c90e83c6-3962-4dd5-97fd-050165a9c846","Type":"ContainerStarted","Data":"062baf4bad5e7e0ec0c7b654a609880d367f2654fea553f2b344a31c78ca1ab7"}
Sep 08 18:25:14.562921 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:14.562533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmh5d" event={"ID":"c90e83c6-3962-4dd5-97fd-050165a9c846","Type":"ContainerStarted","Data":"dea618771220e6dbd5c01d9e2f7859ed037f6256624f7d0bbb7224d2fa0118d2"}
Sep 08 18:25:14.564834 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:14.564809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv" event={"ID":"42917c73-bc39-43cc-8538-d9301838cc0b","Type":"ContainerStarted","Data":"e2ecb758aec8fda0a23d1d607c82546068116d3c1a083cc8f51088103ec345a4"}
Sep 08 18:25:14.564950 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:14.564856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv" event={"ID":"42917c73-bc39-43cc-8538-d9301838cc0b","Type":"ContainerStarted","Data":"c173d0b704e616981851ed996c99defc81dc2777b277c9eaf4a082865d62be33"}
Sep 08 18:25:14.564950 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:14.564883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv" event={"ID":"42917c73-bc39-43cc-8538-d9301838cc0b","Type":"ContainerStarted","Data":"5c92936c32eb9c1b19b2cf577e43fe7ef83f07a5a61b87475c1f82c9e8bbaaed"}
Sep 08 18:25:14.566044 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:14.566022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerStarted","Data":"b40845dc9495c4382e63ad9574a9d48e46c4e7d5efce9a156f389e3b47b078bd"}
Sep 08 18:25:14.585758 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:14.585692    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lmh5d" podStartSLOduration=2.262412444 podStartE2EDuration="3.585674286s" totalImagesPullingTime="1.244889783s" totalInitContainerRuntime="78.372059ms" isStatefulPod=true podCreationTimestamp="2026-09-08 18:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:14.58448866 +0000 UTC m=+96.994798742" watchObservedRunningTime="2026-09-08 18:25:14.585674286 +0000 UTC m=+96.995984400"
Sep 08 18:25:14.607696 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:14.607627    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5b6rv" podStartSLOduration=2.087912486 podStartE2EDuration="3.607604588s" totalImagesPullingTime="1.519692102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:14.606569231 +0000 UTC m=+97.016879337" watchObservedRunningTime="2026-09-08 18:25:14.607604588 +0000 UTC m=+97.017914674"
Sep 08 18:25:15.571195 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.571163    2580 generic.go:356] "Generic (PLEG): container finished" podID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerID="a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2" exitCode=0
Sep 08 18:25:15.571576 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.571252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerDied","Data":"a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2"}
Sep 08 18:25:15.963624 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.963588    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54776484f4-hplz8"]
Sep 08 18:25:15.966930 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.966912    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:15.969818 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.969794    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 08 18:25:15.969949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.969837    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2g57tjflhu8uo\""
Sep 08 18:25:15.969949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.969869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 18:25:15.969949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.969900    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kn4q2\""
Sep 08 18:25:15.970960 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.970942    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 08 18:25:15.971012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.970965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 08 18:25:15.977101 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:15.977082    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54776484f4-hplz8"]
Sep 08 18:25:16.128778 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.128725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-metrics-server-audit-profiles\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.128954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.128787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-audit-log\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.128954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.128831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cp6x\" (UniqueName: \"kubernetes.io/projected/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-kube-api-access-4cp6x\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.128954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.128932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.129067 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.128969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-secret-metrics-server-tls\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.129067 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.129001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-secret-metrics-server-client-certs\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.129152 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.129067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-client-ca-bundle\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.229813 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.229720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-audit-log\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.229813 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.229801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cp6x\" (UniqueName: \"kubernetes.io/projected/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-kube-api-access-4cp6x\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.230024 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.229872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.230024 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.229900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-secret-metrics-server-tls\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.230024 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.229934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-secret-metrics-server-client-certs\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.230024 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.229983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-client-ca-bundle\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.230210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.230029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-metrics-server-audit-profiles\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.230210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.230188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-audit-log\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.231360 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.231334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-metrics-server-audit-profiles\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.232027 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.232000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.234121 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.234099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-client-ca-bundle\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.234217 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.234100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-secret-metrics-server-tls\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.234383 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.234366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-secret-metrics-server-client-certs\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.241241 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.241220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cp6x\" (UniqueName: \"kubernetes.io/projected/6860b446-b8b9-41cf-9e5b-12d7dc1f1514-kube-api-access-4cp6x\") pod \"metrics-server-54776484f4-hplz8\" (UID: \"6860b446-b8b9-41cf-9e5b-12d7dc1f1514\") " pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.277322 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.277284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:16.313626 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.313594    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-666fc978bc-65f6p"]
Sep 08 18:25:16.319813 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.319788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.333653 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.333122    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 18:25:16.339020 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.338189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666fc978bc-65f6p"]
Sep 08 18:25:16.427262 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.427230    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tgnbz"
Sep 08 18:25:16.434697 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.434670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rf2\" (UniqueName: \"kubernetes.io/projected/d9184e21-8342-4a91-894b-428380062f8f-kube-api-access-f5rf2\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.434843 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.434730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-oauth-serving-cert\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.434843 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.434769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-service-ca\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.434843 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.434791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-console-config\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.434843 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.434822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-serving-cert\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.435046 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.434870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-oauth-config\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.435046 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.434899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-trusted-ca-bundle\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536059 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.535982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rf2\" (UniqueName: \"kubernetes.io/projected/d9184e21-8342-4a91-894b-428380062f8f-kube-api-access-f5rf2\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536207 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-oauth-serving-cert\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536207 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-service-ca\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536207 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-console-config\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-serving-cert\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-oauth-config\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-trusted-ca-bundle\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536995 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-oauth-serving-cert\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.536995 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-service-ca\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.537321 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.536988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-console-config\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.537321 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.537132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-trusted-ca-bundle\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.538930 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.538910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-serving-cert\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.539019 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.539010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-oauth-config\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.544271 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.544246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rf2\" (UniqueName: \"kubernetes.io/projected/d9184e21-8342-4a91-894b-428380062f8f-kube-api-access-f5rf2\") pod \"console-666fc978bc-65f6p\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") " pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.556664 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.556649    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54776484f4-hplz8"]
Sep 08 18:25:16.559125 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:25:16.559104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6860b446_b8b9_41cf_9e5b_12d7dc1f1514.slice/crio-439d722362af4bbe3959c02fcb5123b266625992688b88f0d0c8f568aeaa73c3 WatchSource:0}: Error finding container 439d722362af4bbe3959c02fcb5123b266625992688b88f0d0c8f568aeaa73c3: Status 404 returned error can't find the container with id 439d722362af4bbe3959c02fcb5123b266625992688b88f0d0c8f568aeaa73c3
Sep 08 18:25:16.576749 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.576725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54776484f4-hplz8" event={"ID":"6860b446-b8b9-41cf-9e5b-12d7dc1f1514","Type":"ContainerStarted","Data":"439d722362af4bbe3959c02fcb5123b266625992688b88f0d0c8f568aeaa73c3"}
Sep 08 18:25:16.639296 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.639264    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:16.914826 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:16.914805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666fc978bc-65f6p"]
Sep 08 18:25:16.917053 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:25:16.917027    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9184e21_8342_4a91_894b_428380062f8f.slice/crio-bef4cbac238815b2317e365b4307e6b24fda3043df20f4d2ffae8fae4f4dff0d WatchSource:0}: Error finding container bef4cbac238815b2317e365b4307e6b24fda3043df20f4d2ffae8fae4f4dff0d: Status 404 returned error can't find the container with id bef4cbac238815b2317e365b4307e6b24fda3043df20f4d2ffae8fae4f4dff0d
Sep 08 18:25:17.592492 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.592458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fc978bc-65f6p" event={"ID":"d9184e21-8342-4a91-894b-428380062f8f","Type":"ContainerStarted","Data":"af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b"}
Sep 08 18:25:17.592794 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.592556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fc978bc-65f6p" event={"ID":"d9184e21-8342-4a91-894b-428380062f8f","Type":"ContainerStarted","Data":"bef4cbac238815b2317e365b4307e6b24fda3043df20f4d2ffae8fae4f4dff0d"}
Sep 08 18:25:17.596995 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.596969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerStarted","Data":"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239"}
Sep 08 18:25:17.597081 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.597044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerStarted","Data":"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84"}
Sep 08 18:25:17.597118 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.597086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerStarted","Data":"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5"}
Sep 08 18:25:17.614273 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.614222    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-666fc978bc-65f6p" podStartSLOduration=1.614200479 podStartE2EDuration="1.614200479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:17.612197453 +0000 UTC m=+100.022507556" watchObservedRunningTime="2026-09-08 18:25:17.614200479 +0000 UTC m=+100.024510561"
Sep 08 18:25:17.788586 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.788252    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:25:17.792874 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.792851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.795648 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.795626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 18:25:17.795835 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.795814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 18:25:17.795936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.795883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 18:25:17.795936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.795911    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 18:25:17.796098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.795998    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 18:25:17.796098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.795998    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-12ftiic5vahbc\""
Sep 08 18:25:17.796098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.796022    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 18:25:17.796347 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.796330    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 18:25:17.796446 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.796349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 18:25:17.796446 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.796386    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 18:25:17.796566 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.796555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 18:25:17.796620 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.796598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wqnnx\""
Sep 08 18:25:17.797153 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.797138    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 18:25:17.797908 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.797890    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 18:25:17.800354 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.800335    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 18:25:17.803577 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.803558    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 18:25:17.809935 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.809916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:25:17.847627 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847773 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-config-out\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847773 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847773 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847773 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847773 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.847985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.847930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rm6\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-kube-api-access-w6rm6\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848270 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848270 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848270 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848270 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-config\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848270 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-web-config\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.848472 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.848371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949335 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949335 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949556 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-config-out\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949638 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949638 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949847 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949847 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949847 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.949847 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rm6\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-kube-api-access-w6rm6\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.949982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.950006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.950038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-config\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950408 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.950075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-web-config\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.950408 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.950365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.950676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.950839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.951642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.951728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.951881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.952323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.952967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.953100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.953524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.954329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-config-out\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.954496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.954451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.955095 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.954789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.955506 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.955486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-config\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.955642 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.955495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.955843 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.955819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-web-config\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.956091 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.956071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.956479 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.956463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.956817 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.956802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:17.958192 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:17.958176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rm6\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-kube-api-access-w6rm6\") pod \"prometheus-k8s-0\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:18.104821 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:18.104607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:18.603389 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:18.603353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54776484f4-hplz8" event={"ID":"6860b446-b8b9-41cf-9e5b-12d7dc1f1514","Type":"ContainerStarted","Data":"abc030a1540dd34c71f558ce8e0e2f2c26032908e279b9602ccfc39c5eb6ad0d"}
Sep 08 18:25:18.607440 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:18.607391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerStarted","Data":"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7"}
Sep 08 18:25:18.607579 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:18.607484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerStarted","Data":"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c"}
Sep 08 18:25:18.622722 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:18.622674    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54776484f4-hplz8" podStartSLOduration=1.666723502 podStartE2EDuration="3.62265943s" totalImagesPullingTime="1.955935928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:18.621192313 +0000 UTC m=+101.031502419" watchObservedRunningTime="2026-09-08 18:25:18.62265943 +0000 UTC m=+101.032969511"
Sep 08 18:25:18.745504 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:18.744716    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:25:18.749222 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:25:18.749190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94490cbc_c65e_4b1c_a58b_43a217411e70.slice/crio-38634b8aed2abd78509f778e4d25b83163d9d548cd89ef0acf8a6be82bdbe15a WatchSource:0}: Error finding container 38634b8aed2abd78509f778e4d25b83163d9d548cd89ef0acf8a6be82bdbe15a: Status 404 returned error can't find the container with id 38634b8aed2abd78509f778e4d25b83163d9d548cd89ef0acf8a6be82bdbe15a
Sep 08 18:25:19.615678 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:19.615643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerStarted","Data":"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058"}
Sep 08 18:25:19.617196 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:19.617171    2580 generic.go:356] "Generic (PLEG): container finished" podID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885" exitCode=0
Sep 08 18:25:19.617327 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:19.617200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerDied","Data":"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"}
Sep 08 18:25:19.617327 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:19.617268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerStarted","Data":"38634b8aed2abd78509f778e4d25b83163d9d548cd89ef0acf8a6be82bdbe15a"}
Sep 08 18:25:19.644405 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:19.644361    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.33614076 podStartE2EDuration="7.644342922s" totalImagesPullingTime="4.292736199s" totalInitContainerRuntime="15.465963ms" isStatefulPod=false podCreationTimestamp="2026-09-08 18:25:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:19.643688427 +0000 UTC m=+102.053998531" watchObservedRunningTime="2026-09-08 18:25:19.644342922 +0000 UTC m=+102.054653004"
Sep 08 18:25:23.637820 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:23.637781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerStarted","Data":"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"}
Sep 08 18:25:24.645526 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:24.645487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerStarted","Data":"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"}
Sep 08 18:25:26.639545 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.639505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:26.639945 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.639576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:26.644530 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.644508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:26.661383 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.661348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerStarted","Data":"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"}
Sep 08 18:25:26.661667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.661498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerStarted","Data":"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"}
Sep 08 18:25:26.661667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.661544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerStarted","Data":"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"}
Sep 08 18:25:26.661667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.661569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerStarted","Data":"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"}
Sep 08 18:25:26.665498 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.665468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:25:26.691727 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.691666    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.557021616 podStartE2EDuration="9.691646202s" totalImagesPullingTime="6.1092777s" totalInitContainerRuntime="25.346886ms" isStatefulPod=false podCreationTimestamp="2026-09-08 18:25:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:26.688798443 +0000 UTC m=+109.099108527" watchObservedRunningTime="2026-09-08 18:25:26.691646202 +0000 UTC m=+109.101956312"
Sep 08 18:25:26.735776 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:26.735748    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d6df45f77-b82nb"]
Sep 08 18:25:28.104710 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:28.104680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:25:36.278348 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:36.278320    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:36.278748 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:36.278369    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:50.762269 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:50.762236    2580 generic.go:356] "Generic (PLEG): container finished" podID="1bf50b32-0985-4f48-87a0-ef0b751950db" containerID="6212f8bcfc3a730d5a7fa23f9db3e99bfcbf6de5316fe4f90d715de0e4197cac" exitCode=0
Sep 08 18:25:50.762735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:50.762316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f" event={"ID":"1bf50b32-0985-4f48-87a0-ef0b751950db","Type":"ContainerDied","Data":"6212f8bcfc3a730d5a7fa23f9db3e99bfcbf6de5316fe4f90d715de0e4197cac"}
Sep 08 18:25:50.762735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:50.762643    2580 scope.go:117] "RemoveContainer" containerID="6212f8bcfc3a730d5a7fa23f9db3e99bfcbf6de5316fe4f90d715de0e4197cac"
Sep 08 18:25:50.763858 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:50.763831    2580 generic.go:356] "Generic (PLEG): container finished" podID="d10402b8-2af5-478a-9663-b9efdaf54529" containerID="6fadf6bb757d3c8bfc26fef1d4dd98c3e0d3251111ebab68348d4494e57158d8" exitCode=0
Sep 08 18:25:50.763974 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:50.763919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg" event={"ID":"d10402b8-2af5-478a-9663-b9efdaf54529","Type":"ContainerDied","Data":"6fadf6bb757d3c8bfc26fef1d4dd98c3e0d3251111ebab68348d4494e57158d8"}
Sep 08 18:25:50.764238 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:50.764221    2580 scope.go:117] "RemoveContainer" containerID="6fadf6bb757d3c8bfc26fef1d4dd98c3e0d3251111ebab68348d4494e57158d8"
Sep 08 18:25:51.756832 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:51.756796    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d6df45f77-b82nb" podUID="b0fdf782-84b3-4d87-80b4-5a7647b16827" containerName="console" containerID="cri-o://fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241" gracePeriod=15
Sep 08 18:25:51.770168 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:51.770140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mr28f" event={"ID":"1bf50b32-0985-4f48-87a0-ef0b751950db","Type":"ContainerStarted","Data":"6656f3535adf7013789370a34ef0cea9143ec049e9a725cccff4a4a691959470"}
Sep 08 18:25:51.772123 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:51.772102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-747mg" event={"ID":"d10402b8-2af5-478a-9663-b9efdaf54529","Type":"ContainerStarted","Data":"ebfdb783119fb54cef4dd0a6daaddfb5b899079dcbe7f7d561916c84714adf3e"}
Sep 08 18:25:52.004589 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.004566    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:25:52.165074 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-oauth-config\") pod \"b0fdf782-84b3-4d87-80b4-5a7647b16827\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") "
Sep 08 18:25:52.165246 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-service-ca\") pod \"b0fdf782-84b3-4d87-80b4-5a7647b16827\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") "
Sep 08 18:25:52.165246 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-serving-cert\") pod \"b0fdf782-84b3-4d87-80b4-5a7647b16827\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") "
Sep 08 18:25:52.165362 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4gmk\" (UniqueName: \"kubernetes.io/projected/b0fdf782-84b3-4d87-80b4-5a7647b16827-kube-api-access-f4gmk\") pod \"b0fdf782-84b3-4d87-80b4-5a7647b16827\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") "
Sep 08 18:25:52.165362 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-oauth-serving-cert\") pod \"b0fdf782-84b3-4d87-80b4-5a7647b16827\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") "
Sep 08 18:25:52.165545 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-config\") pod \"b0fdf782-84b3-4d87-80b4-5a7647b16827\" (UID: \"b0fdf782-84b3-4d87-80b4-5a7647b16827\") "
Sep 08 18:25:52.165545 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-service-ca" (OuterVolumeSpecName: "service-ca") pod "b0fdf782-84b3-4d87-80b4-5a7647b16827" (UID: "b0fdf782-84b3-4d87-80b4-5a7647b16827"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:25:52.165733 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165716    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:25:52.165795 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165761    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b0fdf782-84b3-4d87-80b4-5a7647b16827" (UID: "b0fdf782-84b3-4d87-80b4-5a7647b16827"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:25:52.165871 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.165847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-config" (OuterVolumeSpecName: "console-config") pod "b0fdf782-84b3-4d87-80b4-5a7647b16827" (UID: "b0fdf782-84b3-4d87-80b4-5a7647b16827"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:25:52.167633 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.167612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b0fdf782-84b3-4d87-80b4-5a7647b16827" (UID: "b0fdf782-84b3-4d87-80b4-5a7647b16827"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:25:52.167726 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.167648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fdf782-84b3-4d87-80b4-5a7647b16827-kube-api-access-f4gmk" (OuterVolumeSpecName: "kube-api-access-f4gmk") pod "b0fdf782-84b3-4d87-80b4-5a7647b16827" (UID: "b0fdf782-84b3-4d87-80b4-5a7647b16827"). InnerVolumeSpecName "kube-api-access-f4gmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:25:52.167831 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.167811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b0fdf782-84b3-4d87-80b4-5a7647b16827" (UID: "b0fdf782-84b3-4d87-80b4-5a7647b16827"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:25:52.266499 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.266464    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gmk\" (UniqueName: \"kubernetes.io/projected/b0fdf782-84b3-4d87-80b4-5a7647b16827-kube-api-access-f4gmk\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:25:52.266499 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.266492    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-oauth-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:25:52.266499 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.266505    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:25:52.266686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.266519    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-oauth-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:25:52.266686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.266530    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0fdf782-84b3-4d87-80b4-5a7647b16827-console-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:25:52.777557 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.777522    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0fdf782-84b3-4d87-80b4-5a7647b16827" containerID="fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241" exitCode=0
Sep 08 18:25:52.777996 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.777579    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6df45f77-b82nb"
Sep 08 18:25:52.777996 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.777611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6df45f77-b82nb" event={"ID":"b0fdf782-84b3-4d87-80b4-5a7647b16827","Type":"ContainerDied","Data":"fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241"}
Sep 08 18:25:52.777996 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.777661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6df45f77-b82nb" event={"ID":"b0fdf782-84b3-4d87-80b4-5a7647b16827","Type":"ContainerDied","Data":"072d8edc44ddd4c94a6150dd0e6755502763aca1038d7fabcb2d2f1d0b481514"}
Sep 08 18:25:52.778252 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.778239    2580 scope.go:117] "RemoveContainer" containerID="fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241"
Sep 08 18:25:52.787039 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.787024    2580 scope.go:117] "RemoveContainer" containerID="fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241"
Sep 08 18:25:52.787329 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:25:52.787310    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241\": container with ID starting with fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241 not found: ID does not exist" containerID="fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241"
Sep 08 18:25:52.787407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.787342    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241"} err="failed to get container status \"fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241\": rpc error: code = NotFound desc = could not find container \"fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241\": container with ID starting with fa7a8a648ca43789df4b58b095052e40bc97c6fd15b89fe5565cdce2c7a35241 not found: ID does not exist"
Sep 08 18:25:52.802853 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.802828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d6df45f77-b82nb"]
Sep 08 18:25:52.806908 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:52.806890    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d6df45f77-b82nb"]
Sep 08 18:25:54.119662 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:54.119631    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0fdf782-84b3-4d87-80b4-5a7647b16827" path="/var/lib/kubelet/pods/b0fdf782-84b3-4d87-80b4-5a7647b16827/volumes"
Sep 08 18:25:56.283768 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:56.283741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:25:56.288146 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:25:56.288123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54776484f4-hplz8"
Sep 08 18:26:18.105774 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:18.105735    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:18.126951 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:18.126927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:18.902304 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:18.902275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:31.862003 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:31.861920    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:26:31.862495 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:31.862256    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy-metric" containerID="cri-o://99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7" gracePeriod=120
Sep 08 18:26:31.862495 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:31.862303    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="config-reloader" containerID="cri-o://147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84" gracePeriod=120
Sep 08 18:26:31.862495 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:31.862294    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy-web" containerID="cri-o://b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239" gracePeriod=120
Sep 08 18:26:31.862495 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:31.862357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="prom-label-proxy" containerID="cri-o://a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058" gracePeriod=120
Sep 08 18:26:31.862495 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:31.862357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy" containerID="cri-o://d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c" gracePeriod=120
Sep 08 18:26:31.862495 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:31.862253    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="alertmanager" containerID="cri-o://89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5" gracePeriod=120
Sep 08 18:26:32.944904 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:32.944871    2580 generic.go:356] "Generic (PLEG): container finished" podID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerID="a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058" exitCode=0
Sep 08 18:26:32.944904 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:32.944901    2580 generic.go:356] "Generic (PLEG): container finished" podID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerID="d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c" exitCode=0
Sep 08 18:26:32.944904 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:32.944909    2580 generic.go:356] "Generic (PLEG): container finished" podID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerID="147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84" exitCode=0
Sep 08 18:26:32.945308 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:32.944917    2580 generic.go:356] "Generic (PLEG): container finished" podID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerID="89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5" exitCode=0
Sep 08 18:26:32.945308 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:32.944942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerDied","Data":"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058"}
Sep 08 18:26:32.945308 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:32.945032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerDied","Data":"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c"}
Sep 08 18:26:32.945308 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:32.945061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerDied","Data":"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84"}
Sep 08 18:26:32.945308 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:32.945085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerDied","Data":"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5"}
Sep 08 18:26:33.121599 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.121571    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:26:33.238407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238374    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-metrics-client-ca\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238400    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-volume\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-web-config\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-out\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238524    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-main-db\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238668    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-main-tls\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238946 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238721    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-cluster-tls-config\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.238946 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.238885    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:33.239135 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.239090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-tls-assets\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.239135 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.239139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-trusted-ca-bundle\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.239350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.239250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.239350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.239282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m46w8\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-kube-api-access-m46w8\") pod \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\" (UID: \"b397ae8f-85d4-4a37-bbf9-01851739b1a2\") "
Sep 08 18:26:33.239350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.239308    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:26:33.239792 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.239738    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-metrics-client-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.239792 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.239760    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-main-db\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.239940 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.239802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:33.242260 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.242233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-kube-api-access-m46w8" (OuterVolumeSpecName: "kube-api-access-m46w8") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "kube-api-access-m46w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:26:33.242637 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.242608    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-volume" (OuterVolumeSpecName: "config-volume") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:33.242768 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.242743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-out" (OuterVolumeSpecName: "config-out") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:26:33.242825 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.242784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:33.242825 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.242798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:33.243208 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.243187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:33.244036 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.244004    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:33.244345 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.244325    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:26:33.246450 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.246404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:33.253926 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.253904    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-web-config" (OuterVolumeSpecName: "web-config") pod "b397ae8f-85d4-4a37-bbf9-01851739b1a2" (UID: "b397ae8f-85d4-4a37-bbf9-01851739b1a2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:33.341203 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341161    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341203 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341197    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46w8\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-kube-api-access-m46w8\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341212    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341225    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341236    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-web-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341247    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341258    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b397ae8f-85d4-4a37-bbf9-01851739b1a2-config-out\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341269    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-secret-alertmanager-main-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341280    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b397ae8f-85d4-4a37-bbf9-01851739b1a2-cluster-tls-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341291    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b397ae8f-85d4-4a37-bbf9-01851739b1a2-tls-assets\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.341407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.341304    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b397ae8f-85d4-4a37-bbf9-01851739b1a2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:33.953467 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.953412    2580 generic.go:356] "Generic (PLEG): container finished" podID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerID="99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7" exitCode=0
Sep 08 18:26:33.953467 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.953460    2580 generic.go:356] "Generic (PLEG): container finished" podID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerID="b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239" exitCode=0
Sep 08 18:26:33.953911 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.953465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerDied","Data":"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7"}
Sep 08 18:26:33.953911 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.953512    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 18:26:33.953911 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.953541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerDied","Data":"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239"}
Sep 08 18:26:33.953911 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.953577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b397ae8f-85d4-4a37-bbf9-01851739b1a2","Type":"ContainerDied","Data":"b40845dc9495c4382e63ad9574a9d48e46c4e7d5efce9a156f389e3b47b078bd"}
Sep 08 18:26:33.953911 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.953606    2580 scope.go:117] "RemoveContainer" containerID="a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058"
Sep 08 18:26:33.966746 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.962909    2580 scope.go:117] "RemoveContainer" containerID="99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7"
Sep 08 18:26:33.974806 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.974783    2580 scope.go:117] "RemoveContainer" containerID="d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c"
Sep 08 18:26:33.979661 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.979636    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:26:33.984740 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.984687    2580 scope.go:117] "RemoveContainer" containerID="b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239"
Sep 08 18:26:33.985234 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.985209    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 18:26:33.992033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.992011    2580 scope.go:117] "RemoveContainer" containerID="147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84"
Sep 08 18:26:33.999397 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:33.999378    2580 scope.go:117] "RemoveContainer" containerID="89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5"
Sep 08 18:26:34.007330 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.007308    2580 scope.go:117] "RemoveContainer" containerID="a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2"
Sep 08 18:26:34.015000 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.014979    2580 scope.go:117] "RemoveContainer" containerID="a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058"
Sep 08 18:26:34.015337 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:34.015310    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058\": container with ID starting with a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058 not found: ID does not exist" containerID="a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058"
Sep 08 18:26:34.015477 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.015350    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058"} err="failed to get container status \"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058\": rpc error: code = NotFound desc = could not find container \"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058\": container with ID starting with a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058 not found: ID does not exist"
Sep 08 18:26:34.015477 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.015376    2580 scope.go:117] "RemoveContainer" containerID="99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7"
Sep 08 18:26:34.015713 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:34.015686    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7\": container with ID starting with 99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7 not found: ID does not exist" containerID="99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7"
Sep 08 18:26:34.015750 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.015719    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7"} err="failed to get container status \"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7\": rpc error: code = NotFound desc = could not find container \"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7\": container with ID starting with 99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7 not found: ID does not exist"
Sep 08 18:26:34.015750 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.015734    2580 scope.go:117] "RemoveContainer" containerID="d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c"
Sep 08 18:26:34.015984 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:34.015967    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c\": container with ID starting with d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c not found: ID does not exist" containerID="d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c"
Sep 08 18:26:34.016028 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.015988    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c"} err="failed to get container status \"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c\": rpc error: code = NotFound desc = could not find container \"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c\": container with ID starting with d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c not found: ID does not exist"
Sep 08 18:26:34.016028 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.016003    2580 scope.go:117] "RemoveContainer" containerID="b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239"
Sep 08 18:26:34.016239 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:34.016226    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239\": container with ID starting with b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239 not found: ID does not exist" containerID="b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239"
Sep 08 18:26:34.016281 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.016241    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239"} err="failed to get container status \"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239\": rpc error: code = NotFound desc = could not find container \"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239\": container with ID starting with b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239 not found: ID does not exist"
Sep 08 18:26:34.016281 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.016251    2580 scope.go:117] "RemoveContainer" containerID="147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84"
Sep 08 18:26:34.016474 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:34.016457    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84\": container with ID starting with 147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84 not found: ID does not exist" containerID="147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84"
Sep 08 18:26:34.016523 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.016479    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84"} err="failed to get container status \"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84\": rpc error: code = NotFound desc = could not find container \"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84\": container with ID starting with 147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84 not found: ID does not exist"
Sep 08 18:26:34.016523 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.016493    2580 scope.go:117] "RemoveContainer" containerID="89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5"
Sep 08 18:26:34.016745 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:34.016727    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5\": container with ID starting with 89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5 not found: ID does not exist" containerID="89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5"
Sep 08 18:26:34.016854 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.016752    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5"} err="failed to get container status \"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5\": rpc error: code = NotFound desc = could not find container \"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5\": container with ID starting with 89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5 not found: ID does not exist"
Sep 08 18:26:34.016854 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.016773    2580 scope.go:117] "RemoveContainer" containerID="a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2"
Sep 08 18:26:34.017022 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:34.017006    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2\": container with ID starting with a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2 not found: ID does not exist" containerID="a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2"
Sep 08 18:26:34.017060 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017025    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2"} err="failed to get container status \"a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2\": rpc error: code = NotFound desc = could not find container \"a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2\": container with ID starting with a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2 not found: ID does not exist"
Sep 08 18:26:34.017060 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017040    2580 scope.go:117] "RemoveContainer" containerID="a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058"
Sep 08 18:26:34.017251 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058"} err="failed to get container status \"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058\": rpc error: code = NotFound desc = could not find container \"a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058\": container with ID starting with a07e95486ae5a98cbf39d96a852c0521d1c16febe3518e25bdde137b434d3058 not found: ID does not exist"
Sep 08 18:26:34.017320 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017253    2580 scope.go:117] "RemoveContainer" containerID="99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7"
Sep 08 18:26:34.017521 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017504    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7"} err="failed to get container status \"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7\": rpc error: code = NotFound desc = could not find container \"99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7\": container with ID starting with 99adba22b20ca77a3f3929ae70351e8c22fff23815415fce7af4ee5d63f2c9a7 not found: ID does not exist"
Sep 08 18:26:34.017572 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017522    2580 scope.go:117] "RemoveContainer" containerID="d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c"
Sep 08 18:26:34.017758 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017741    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c"} err="failed to get container status \"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c\": rpc error: code = NotFound desc = could not find container \"d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c\": container with ID starting with d1685ccc7d03ca48f63cdbd86feb909774c779de7096cfa77890df43bc2d819c not found: ID does not exist"
Sep 08 18:26:34.017804 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017759    2580 scope.go:117] "RemoveContainer" containerID="b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239"
Sep 08 18:26:34.017990 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239"} err="failed to get container status \"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239\": rpc error: code = NotFound desc = could not find container \"b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239\": container with ID starting with b5063c609e69f07b2e6a50f1cf0f62d00f6f56d61d5902d2660059575528c239 not found: ID does not exist"
Sep 08 18:26:34.018035 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.017991    2580 scope.go:117] "RemoveContainer" containerID="147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84"
Sep 08 18:26:34.018209 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.018190    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84"} err="failed to get container status \"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84\": rpc error: code = NotFound desc = could not find container \"147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84\": container with ID starting with 147c89ceb4c2d91ec901078fc207b4ce203cce3ec34e7d5da97719f6dcd8da84 not found: ID does not exist"
Sep 08 18:26:34.018276 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.018210    2580 scope.go:117] "RemoveContainer" containerID="89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5"
Sep 08 18:26:34.018475 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.018450    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5"} err="failed to get container status \"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5\": rpc error: code = NotFound desc = could not find container \"89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5\": container with ID starting with 89c0aff60a8d2396ed8d0a743c377a35c9bd6eed070ba6f4c1a228f0c4a31bc5 not found: ID does not exist"
Sep 08 18:26:34.018527 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.018476    2580 scope.go:117] "RemoveContainer" containerID="a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2"
Sep 08 18:26:34.018682 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.018665    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2"} err="failed to get container status \"a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2\": rpc error: code = NotFound desc = could not find container \"a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2\": container with ID starting with a6e81e8c8231858fd55af2378a911b16afa8b332fd692781345eef238d3b2ba2 not found: ID does not exist"
Sep 08 18:26:34.120137 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:34.120109    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" path="/var/lib/kubelet/pods/b397ae8f-85d4-4a37-bbf9-01851739b1a2/volumes"
Sep 08 18:26:36.200094 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.200062    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:26:36.200539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.200335    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="prometheus" containerID="cri-o://3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa" gracePeriod=600
Sep 08 18:26:36.200539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.200379    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy" containerID="cri-o://541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2" gracePeriod=600
Sep 08 18:26:36.200539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.200398    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="thanos-sidecar" containerID="cri-o://50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576" gracePeriod=600
Sep 08 18:26:36.200539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.200457    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy-web" containerID="cri-o://b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9" gracePeriod=600
Sep 08 18:26:36.200539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.200474    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="config-reloader" containerID="cri-o://7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458" gracePeriod=600
Sep 08 18:26:36.200809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.200457    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1" gracePeriod=600
Sep 08 18:26:36.439131 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.439109    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:36.571544 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571447    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-config-out\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571544 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571497    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-tls\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571544 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-thanos-prometheus-http-client-file\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-db\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-kube-rbac-proxy\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-1\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-web-config\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-tls-assets\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-serving-certs-ca-bundle\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571749    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-2\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.571809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-trusted-ca-bundle\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571829    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-metrics-client-ca\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-0\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571932    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-kubelet-serving-ca-bundle\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6rm6\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-kube-api-access-w6rm6\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.571997    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.572021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-config\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.572054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-metrics-client-certs\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.572081    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-grpc-tls\") pod \"94490cbc-c65e-4b1c-a58b-43a217411e70\" (UID: \"94490cbc-c65e-4b1c-a58b-43a217411e70\") "
Sep 08 18:26:36.572186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.572120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:36.572738 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.572355    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.572945 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.572918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:36.573713 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.573628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:26:36.573883 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.573837    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:36.574180 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.574151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:36.574600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.574574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:36.574719 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.574687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.575195 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.575169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:36.575356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.575328    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:26:36.576188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.576071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.576188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.576097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.576188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.576141    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-config-out" (OuterVolumeSpecName: "config-out") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:26:36.576188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.576154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.576464 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.576356    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:26:36.576537 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.576479    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.576866 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.576843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.576944 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.576871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.577374 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.577355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-kube-api-access-w6rm6" (OuterVolumeSpecName: "kube-api-access-w6rm6") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "kube-api-access-w6rm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:26:36.577764 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.577747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-config" (OuterVolumeSpecName: "config") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.586656 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.586633    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-web-config" (OuterVolumeSpecName: "web-config") pod "94490cbc-c65e-4b1c-a58b-43a217411e70" (UID: "94490cbc-c65e-4b1c-a58b-43a217411e70"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:26:36.673561 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673530    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673561 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673558    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673572    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673585    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rm6\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-kube-api-access-w6rm6\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673597    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673608    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673621    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-metrics-client-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673635    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-grpc-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673647    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-config-out\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673657    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673670    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673691    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-db\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673704    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-secret-kube-rbac-proxy\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673715    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94490cbc-c65e-4b1c-a58b-43a217411e70-web-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673726    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94490cbc-c65e-4b1c-a58b-43a217411e70-tls-assets\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673737    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673750    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673762    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.673875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.673773    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94490cbc-c65e-4b1c-a58b-43a217411e70-configmap-metrics-client-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:26:36.969978 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.969878    2580 generic.go:356] "Generic (PLEG): container finished" podID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1" exitCode=0
Sep 08 18:26:36.969978 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.969936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerDied","Data":"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"}
Sep 08 18:26:36.969978 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.969913    2580 generic.go:356] "Generic (PLEG): container finished" podID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2" exitCode=0
Sep 08 18:26:36.969978 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.969984    2580 generic.go:356] "Generic (PLEG): container finished" podID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9" exitCode=0
Sep 08 18:26:36.970306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.969996    2580 generic.go:356] "Generic (PLEG): container finished" podID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576" exitCode=0
Sep 08 18:26:36.970306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970005    2580 generic.go:356] "Generic (PLEG): container finished" podID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458" exitCode=0
Sep 08 18:26:36.970306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970013    2580 generic.go:356] "Generic (PLEG): container finished" podID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa" exitCode=0
Sep 08 18:26:36.970306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerDied","Data":"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"}
Sep 08 18:26:36.970306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerDied","Data":"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"}
Sep 08 18:26:36.970306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerDied","Data":"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"}
Sep 08 18:26:36.970306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerDied","Data":"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"}
Sep 08 18:26:36.970306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerDied","Data":"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"}
Sep 08 18:26:36.970685 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970341    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:36.970685 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"94490cbc-c65e-4b1c-a58b-43a217411e70","Type":"ContainerDied","Data":"38634b8aed2abd78509f778e4d25b83163d9d548cd89ef0acf8a6be82bdbe15a"}
Sep 08 18:26:36.970685 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.970413    2580 scope.go:117] "RemoveContainer" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"
Sep 08 18:26:36.984444 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.984414    2580 scope.go:117] "RemoveContainer" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"
Sep 08 18:26:36.991379 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.991364    2580 scope.go:117] "RemoveContainer" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"
Sep 08 18:26:36.998179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.998160    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:26:36.998637 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:36.998622    2580 scope.go:117] "RemoveContainer" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"
Sep 08 18:26:37.003220 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.003196    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:26:37.005683 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.005669    2580 scope.go:117] "RemoveContainer" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"
Sep 08 18:26:37.014167 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.014150    2580 scope.go:117] "RemoveContainer" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"
Sep 08 18:26:37.021262 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.021246    2580 scope.go:117] "RemoveContainer" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"
Sep 08 18:26:37.028138 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028115    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:26:37.028390 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028379    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy-web"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028392    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy-web"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028401    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="init-config-reloader"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028407    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="init-config-reloader"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028414    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy-thanos"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028419    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy-thanos"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028445    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028454    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028463    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="config-reloader"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028469    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="config-reloader"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028476    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="prom-label-proxy"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028482    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="prom-label-proxy"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028490    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="init-config-reloader"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028497    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="init-config-reloader"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028505    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy-metric"
Sep 08 18:26:37.028503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028510    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy-metric"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028518    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="prometheus"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028523    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="prometheus"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028529    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028533    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028539    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="alertmanager"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028544    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="alertmanager"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028552    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0fdf782-84b3-4d87-80b4-5a7647b16827" containerName="console"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028557    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0fdf782-84b3-4d87-80b4-5a7647b16827" containerName="console"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028563    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy-web"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028567    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy-web"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028574    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="thanos-sidecar"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028579    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="thanos-sidecar"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028586    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="config-reloader"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028591    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="config-reloader"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028638    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy-metric"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028646    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="config-reloader"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028655    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="prom-label-proxy"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028661    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy-web"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028667    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0fdf782-84b3-4d87-80b4-5a7647b16827" containerName="console"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028673    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy-web"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028678    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="alertmanager"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028683    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="kube-rbac-proxy"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028689    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="thanos-sidecar"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028694    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b397ae8f-85d4-4a37-bbf9-01851739b1a2" containerName="config-reloader"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028698    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028703    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="kube-rbac-proxy-thanos"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028708    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" containerName="prometheus"
Sep 08 18:26:37.028936 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.028882    2580 scope.go:117] "RemoveContainer" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:37.029179    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": container with ID starting with ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1 not found: ID does not exist" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.029203    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"} err="failed to get container status \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": rpc error: code = NotFound desc = could not find container \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": container with ID starting with ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1 not found: ID does not exist"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.029227    2580 scope.go:117] "RemoveContainer" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:37.029533    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": container with ID starting with 541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2 not found: ID does not exist" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.029554    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"} err="failed to get container status \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": rpc error: code = NotFound desc = could not find container \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": container with ID starting with 541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2 not found: ID does not exist"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.029570    2580 scope.go:117] "RemoveContainer" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:37.029827    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": container with ID starting with b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9 not found: ID does not exist" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.029844    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"} err="failed to get container status \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": rpc error: code = NotFound desc = could not find container \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": container with ID starting with b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9 not found: ID does not exist"
Sep 08 18:26:37.029957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.029856    2580 scope.go:117] "RemoveContainer" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"
Sep 08 18:26:37.030273 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:37.030076    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": container with ID starting with 50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576 not found: ID does not exist" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"
Sep 08 18:26:37.030273 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.030091    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"} err="failed to get container status \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": rpc error: code = NotFound desc = could not find container \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": container with ID starting with 50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576 not found: ID does not exist"
Sep 08 18:26:37.030273 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.030102    2580 scope.go:117] "RemoveContainer" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"
Sep 08 18:26:37.030396 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:37.030378    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": container with ID starting with 7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458 not found: ID does not exist" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"
Sep 08 18:26:37.030458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.030400    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"} err="failed to get container status \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": rpc error: code = NotFound desc = could not find container \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": container with ID starting with 7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458 not found: ID does not exist"
Sep 08 18:26:37.030458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.030413    2580 scope.go:117] "RemoveContainer" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"
Sep 08 18:26:37.030687 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:37.030672    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": container with ID starting with 3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa not found: ID does not exist" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"
Sep 08 18:26:37.030731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.030693    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"} err="failed to get container status \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": rpc error: code = NotFound desc = could not find container \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": container with ID starting with 3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa not found: ID does not exist"
Sep 08 18:26:37.030731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.030708    2580 scope.go:117] "RemoveContainer" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"
Sep 08 18:26:37.030947 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:26:37.030931    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": container with ID starting with f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885 not found: ID does not exist" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"
Sep 08 18:26:37.030992 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.030950    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"} err="failed to get container status \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": rpc error: code = NotFound desc = could not find container \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": container with ID starting with f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885 not found: ID does not exist"
Sep 08 18:26:37.030992 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.030963    2580 scope.go:117] "RemoveContainer" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"
Sep 08 18:26:37.031184 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.031167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"} err="failed to get container status \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": rpc error: code = NotFound desc = could not find container \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": container with ID starting with ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1 not found: ID does not exist"
Sep 08 18:26:37.031224 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.031185    2580 scope.go:117] "RemoveContainer" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"
Sep 08 18:26:37.031405 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.031388    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"} err="failed to get container status \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": rpc error: code = NotFound desc = could not find container \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": container with ID starting with 541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2 not found: ID does not exist"
Sep 08 18:26:37.031462 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.031406    2580 scope.go:117] "RemoveContainer" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"
Sep 08 18:26:37.031808 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.031788    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"} err="failed to get container status \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": rpc error: code = NotFound desc = could not find container \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": container with ID starting with b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9 not found: ID does not exist"
Sep 08 18:26:37.031890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.031811    2580 scope.go:117] "RemoveContainer" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"
Sep 08 18:26:37.032104 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.032087    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"} err="failed to get container status \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": rpc error: code = NotFound desc = could not find container \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": container with ID starting with 50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576 not found: ID does not exist"
Sep 08 18:26:37.032151 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.032105    2580 scope.go:117] "RemoveContainer" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"
Sep 08 18:26:37.032348 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.032330    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"} err="failed to get container status \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": rpc error: code = NotFound desc = could not find container \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": container with ID starting with 7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458 not found: ID does not exist"
Sep 08 18:26:37.032389 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.032349    2580 scope.go:117] "RemoveContainer" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"
Sep 08 18:26:37.032609 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.032593    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"} err="failed to get container status \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": rpc error: code = NotFound desc = could not find container \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": container with ID starting with 3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa not found: ID does not exist"
Sep 08 18:26:37.032682 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.032609    2580 scope.go:117] "RemoveContainer" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"
Sep 08 18:26:37.032828 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.032813    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"} err="failed to get container status \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": rpc error: code = NotFound desc = could not find container \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": container with ID starting with f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885 not found: ID does not exist"
Sep 08 18:26:37.032870 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.032829    2580 scope.go:117] "RemoveContainer" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"
Sep 08 18:26:37.033042 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033024    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"} err="failed to get container status \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": rpc error: code = NotFound desc = could not find container \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": container with ID starting with ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1 not found: ID does not exist"
Sep 08 18:26:37.033083 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033043    2580 scope.go:117] "RemoveContainer" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"
Sep 08 18:26:37.033270 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033243    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"} err="failed to get container status \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": rpc error: code = NotFound desc = could not find container \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": container with ID starting with 541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2 not found: ID does not exist"
Sep 08 18:26:37.033346 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033272    2580 scope.go:117] "RemoveContainer" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"
Sep 08 18:26:37.033530 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.033617 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033514    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"} err="failed to get container status \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": rpc error: code = NotFound desc = could not find container \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": container with ID starting with b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9 not found: ID does not exist"
Sep 08 18:26:37.033670 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033622    2580 scope.go:117] "RemoveContainer" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"
Sep 08 18:26:37.033983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033956    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"} err="failed to get container status \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": rpc error: code = NotFound desc = could not find container \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": container with ID starting with 50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576 not found: ID does not exist"
Sep 08 18:26:37.033983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.033983    2580 scope.go:117] "RemoveContainer" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"
Sep 08 18:26:37.034279 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.034259    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"} err="failed to get container status \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": rpc error: code = NotFound desc = could not find container \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": container with ID starting with 7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458 not found: ID does not exist"
Sep 08 18:26:37.034330 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.034280    2580 scope.go:117] "RemoveContainer" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"
Sep 08 18:26:37.034613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.034591    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"} err="failed to get container status \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": rpc error: code = NotFound desc = could not find container \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": container with ID starting with 3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa not found: ID does not exist"
Sep 08 18:26:37.034613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.034611    2580 scope.go:117] "RemoveContainer" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"
Sep 08 18:26:37.034847 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.034832    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"} err="failed to get container status \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": rpc error: code = NotFound desc = could not find container \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": container with ID starting with f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885 not found: ID does not exist"
Sep 08 18:26:37.034847 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.034847    2580 scope.go:117] "RemoveContainer" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"
Sep 08 18:26:37.035084 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.035063    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"} err="failed to get container status \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": rpc error: code = NotFound desc = could not find container \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": container with ID starting with ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1 not found: ID does not exist"
Sep 08 18:26:37.035134 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.035085    2580 scope.go:117] "RemoveContainer" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"
Sep 08 18:26:37.035327 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.035311    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"} err="failed to get container status \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": rpc error: code = NotFound desc = could not find container \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": container with ID starting with 541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2 not found: ID does not exist"
Sep 08 18:26:37.035370 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.035327    2580 scope.go:117] "RemoveContainer" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"
Sep 08 18:26:37.035621 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.035604    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"} err="failed to get container status \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": rpc error: code = NotFound desc = could not find container \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": container with ID starting with b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9 not found: ID does not exist"
Sep 08 18:26:37.035673 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.035621    2580 scope.go:117] "RemoveContainer" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"
Sep 08 18:26:37.035866 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.035848    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"} err="failed to get container status \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": rpc error: code = NotFound desc = could not find container \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": container with ID starting with 50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576 not found: ID does not exist"
Sep 08 18:26:37.035944 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.035868    2580 scope.go:117] "RemoveContainer" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"
Sep 08 18:26:37.036157 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036136    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"} err="failed to get container status \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": rpc error: code = NotFound desc = could not find container \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": container with ID starting with 7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458 not found: ID does not exist"
Sep 08 18:26:37.036243 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036158    2580 scope.go:117] "RemoveContainer" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"
Sep 08 18:26:37.036515 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-12ftiic5vahbc\""
Sep 08 18:26:37.036515 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036490    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 18:26:37.036686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036511    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"} err="failed to get container status \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": rpc error: code = NotFound desc = could not find container \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": container with ID starting with 3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa not found: ID does not exist"
Sep 08 18:26:37.036686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036531    2580 scope.go:117] "RemoveContainer" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"
Sep 08 18:26:37.036686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036416    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 18:26:37.036686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036581    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 18:26:37.036686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036661    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 18:26:37.036954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 18:26:37.036954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036807    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"} err="failed to get container status \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": rpc error: code = NotFound desc = could not find container \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": container with ID starting with f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885 not found: ID does not exist"
Sep 08 18:26:37.036954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036828    2580 scope.go:117] "RemoveContainer" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"
Sep 08 18:26:37.037057 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.036986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 18:26:37.037057 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037044    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 18:26:37.037187 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037173    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 18:26:37.037241 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 18:26:37.037241 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037220    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"} err="failed to get container status \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": rpc error: code = NotFound desc = could not find container \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": container with ID starting with ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1 not found: ID does not exist"
Sep 08 18:26:37.037314 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037243    2580 scope.go:117] "RemoveContainer" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"
Sep 08 18:26:37.037379 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037344    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 18:26:37.037468 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wqnnx\""
Sep 08 18:26:37.037552 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037524    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"} err="failed to get container status \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": rpc error: code = NotFound desc = could not find container \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": container with ID starting with 541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2 not found: ID does not exist"
Sep 08 18:26:37.037552 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037548    2580 scope.go:117] "RemoveContainer" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"
Sep 08 18:26:37.037662 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 18:26:37.037662 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037604    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 18:26:37.037956 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037938    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"} err="failed to get container status \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": rpc error: code = NotFound desc = could not find container \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": container with ID starting with b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9 not found: ID does not exist"
Sep 08 18:26:37.038021 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.037957    2580 scope.go:117] "RemoveContainer" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"
Sep 08 18:26:37.038234 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.038210    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"} err="failed to get container status \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": rpc error: code = NotFound desc = could not find container \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": container with ID starting with 50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576 not found: ID does not exist"
Sep 08 18:26:37.038234 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.038235    2580 scope.go:117] "RemoveContainer" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"
Sep 08 18:26:37.038489 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.038466    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"} err="failed to get container status \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": rpc error: code = NotFound desc = could not find container \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": container with ID starting with 7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458 not found: ID does not exist"
Sep 08 18:26:37.038577 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.038490    2580 scope.go:117] "RemoveContainer" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"
Sep 08 18:26:37.038759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.038734    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"} err="failed to get container status \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": rpc error: code = NotFound desc = could not find container \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": container with ID starting with 3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa not found: ID does not exist"
Sep 08 18:26:37.038759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.038756    2580 scope.go:117] "RemoveContainer" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"
Sep 08 18:26:37.039063 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.039035    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"} err="failed to get container status \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": rpc error: code = NotFound desc = could not find container \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": container with ID starting with f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885 not found: ID does not exist"
Sep 08 18:26:37.039155 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.039066    2580 scope.go:117] "RemoveContainer" containerID="ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"
Sep 08 18:26:37.039501 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.039478    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1"} err="failed to get container status \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": rpc error: code = NotFound desc = could not find container \"ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1\": container with ID starting with ca2c3f3cc5382a7274aaee3878fbf5b2d26070ea41260965121019d90e8548d1 not found: ID does not exist"
Sep 08 18:26:37.039593 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.039502    2580 scope.go:117] "RemoveContainer" containerID="541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"
Sep 08 18:26:37.039803 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.039780    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2"} err="failed to get container status \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": rpc error: code = NotFound desc = could not find container \"541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2\": container with ID starting with 541727a317e46a236d17e981d2cf67d07b070f21aa18618476c1363548c687c2 not found: ID does not exist"
Sep 08 18:26:37.040010 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.039805    2580 scope.go:117] "RemoveContainer" containerID="b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"
Sep 08 18:26:37.040089 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.040057    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9"} err="failed to get container status \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": rpc error: code = NotFound desc = could not find container \"b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9\": container with ID starting with b02e6ebc90c3a2dd7453c165e41b1629b8832fa103907acc960b76798f77f1b9 not found: ID does not exist"
Sep 08 18:26:37.040089 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.040072    2580 scope.go:117] "RemoveContainer" containerID="50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"
Sep 08 18:26:37.040545 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.040307    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576"} err="failed to get container status \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": rpc error: code = NotFound desc = could not find container \"50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576\": container with ID starting with 50f1bd1acecc2a6bcc2948d88e8cd9b2090872207905e47169594a548e1c2576 not found: ID does not exist"
Sep 08 18:26:37.040545 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.040328    2580 scope.go:117] "RemoveContainer" containerID="7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"
Sep 08 18:26:37.040696 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.040566    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 18:26:37.040753 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.040729    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458"} err="failed to get container status \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": rpc error: code = NotFound desc = could not find container \"7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458\": container with ID starting with 7c631ee29bb6b413a01581fa678f42c4c3eafc77f1b50b97f3ef60bec9757458 not found: ID does not exist"
Sep 08 18:26:37.040804 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.040751    2580 scope.go:117] "RemoveContainer" containerID="3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"
Sep 08 18:26:37.041065 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.041029    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa"} err="failed to get container status \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": rpc error: code = NotFound desc = could not find container \"3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa\": container with ID starting with 3897b0e2b40fc1adabbaa1427076cc7c37f78825ed24158dbbee596d9ac814fa not found: ID does not exist"
Sep 08 18:26:37.041065 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.041051    2580 scope.go:117] "RemoveContainer" containerID="f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"
Sep 08 18:26:37.041389 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.041359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885"} err="failed to get container status \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": rpc error: code = NotFound desc = could not find container \"f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885\": container with ID starting with f4f6b749694aa5fe42ed7b89c0717156ee487ad366a16c754df4592db1f95885 not found: ID does not exist"
Sep 08 18:26:37.041902 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.041886    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 18:26:37.048892 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.048871    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:26:37.177541 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.177735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.177735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.177735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.177735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.177735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.177735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.177735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90269e2-ebdb-417d-9071-da0467a09274-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.177735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-web-config\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-config\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90269e2-ebdb-417d-9071-da0467a09274-config-out\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczsg\" (UniqueName: \"kubernetes.io/projected/d90269e2-ebdb-417d-9071-da0467a09274-kube-api-access-tczsg\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.178032 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.177966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.278506 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.278506 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.278506 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90269e2-ebdb-417d-9071-da0467a09274-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-web-config\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-config\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90269e2-ebdb-417d-9071-da0467a09274-config-out\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczsg\" (UniqueName: \"kubernetes.io/projected/d90269e2-ebdb-417d-9071-da0467a09274-kube-api-access-tczsg\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.278966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279033 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.279000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.279218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.279305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.279400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.279414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.279759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.279599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.281771 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.280449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.282231 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.281952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.282231 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.282026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-web-config\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.282231 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.282099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.283715 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.282737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90269e2-ebdb-417d-9071-da0467a09274-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.283715 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.282776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.283715 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.283241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90269e2-ebdb-417d-9071-da0467a09274-config-out\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.283715 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.283358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.283897 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.283758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d90269e2-ebdb-417d-9071-da0467a09274-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.284288 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.284267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.284555 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.284534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.284722 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.284704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.285404 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.285383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.285495 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.285406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d90269e2-ebdb-417d-9071-da0467a09274-config\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.291333 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.291314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczsg\" (UniqueName: \"kubernetes.io/projected/d90269e2-ebdb-417d-9071-da0467a09274-kube-api-access-tczsg\") pod \"prometheus-k8s-0\" (UID: \"d90269e2-ebdb-417d-9071-da0467a09274\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.344379 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.344342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:37.624772 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.624752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 18:26:37.626970 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:26:37.626948    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90269e2_ebdb_417d_9071_da0467a09274.slice/crio-4a6060e721be45db197e1a8835b165e1f2737eb150965c6e10875a8ce07af231 WatchSource:0}: Error finding container 4a6060e721be45db197e1a8835b165e1f2737eb150965c6e10875a8ce07af231: Status 404 returned error can't find the container with id 4a6060e721be45db197e1a8835b165e1f2737eb150965c6e10875a8ce07af231
Sep 08 18:26:37.977038 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.976944    2580 generic.go:356] "Generic (PLEG): container finished" podID="d90269e2-ebdb-417d-9071-da0467a09274" containerID="d6ccd5e8b29dceb6dc0f365233df11f821cbe70d09296c509e89353d3cd79553" exitCode=0
Sep 08 18:26:37.977038 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.976992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90269e2-ebdb-417d-9071-da0467a09274","Type":"ContainerDied","Data":"d6ccd5e8b29dceb6dc0f365233df11f821cbe70d09296c509e89353d3cd79553"}
Sep 08 18:26:37.977215 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:37.977051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90269e2-ebdb-417d-9071-da0467a09274","Type":"ContainerStarted","Data":"4a6060e721be45db197e1a8835b165e1f2737eb150965c6e10875a8ce07af231"}
Sep 08 18:26:38.124767 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:38.124735    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94490cbc-c65e-4b1c-a58b-43a217411e70" path="/var/lib/kubelet/pods/94490cbc-c65e-4b1c-a58b-43a217411e70/volumes"
Sep 08 18:26:38.983540 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:38.983511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90269e2-ebdb-417d-9071-da0467a09274","Type":"ContainerStarted","Data":"4fe7f0bc638d33afd5344453bf173fdd5443fe43cd72d400f1315abdc4f1ac49"}
Sep 08 18:26:38.983927 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:38.983568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90269e2-ebdb-417d-9071-da0467a09274","Type":"ContainerStarted","Data":"5b410153c9d60e3901c7ea708d0b4a3073ef0195086c44887b39c1e2420debbf"}
Sep 08 18:26:38.983927 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:38.983596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90269e2-ebdb-417d-9071-da0467a09274","Type":"ContainerStarted","Data":"eea456de1e663de246f7a99bc48e3ccbb4407336bf06a37b2746dae65b4c761f"}
Sep 08 18:26:38.983927 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:38.983621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90269e2-ebdb-417d-9071-da0467a09274","Type":"ContainerStarted","Data":"48a376abb771f9ca3f9461308d34b082e191f1fcca83ee4424a75a1a65502287"}
Sep 08 18:26:38.983927 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:38.983645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90269e2-ebdb-417d-9071-da0467a09274","Type":"ContainerStarted","Data":"8650c0596b4f1488a2074a2671fa0b2976dc9d7a8d58944b2f819dfb83f3ad58"}
Sep 08 18:26:38.983927 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:38.983669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90269e2-ebdb-417d-9071-da0467a09274","Type":"ContainerStarted","Data":"a755433d5a70b79552073d8f8d3a3c50561236c4e676a175be8d837211bcb7b4"}
Sep 08 18:26:39.013656 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:39.013601    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.995291596 podStartE2EDuration="2.013579972s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.288376ms" isStatefulPod=false podCreationTimestamp="2026-09-08 18:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:26:39.011093325 +0000 UTC m=+181.421403429" watchObservedRunningTime="2026-09-08 18:26:39.013579972 +0000 UTC m=+181.423890053"
Sep 08 18:26:42.345378 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:42.345332    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:26:50.280529 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:26:50.280496    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666fc978bc-65f6p"]
Sep 08 18:27:15.307972 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.307934    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-666fc978bc-65f6p" podUID="d9184e21-8342-4a91-894b-428380062f8f" containerName="console" containerID="cri-o://af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b" gracePeriod=15
Sep 08 18:27:15.542116 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.542093    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:27:15.704365 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-service-ca\") pod \"d9184e21-8342-4a91-894b-428380062f8f\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") "
Sep 08 18:27:15.704595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704384    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-oauth-serving-cert\") pod \"d9184e21-8342-4a91-894b-428380062f8f\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") "
Sep 08 18:27:15.704595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704405    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5rf2\" (UniqueName: \"kubernetes.io/projected/d9184e21-8342-4a91-894b-428380062f8f-kube-api-access-f5rf2\") pod \"d9184e21-8342-4a91-894b-428380062f8f\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") "
Sep 08 18:27:15.704595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-trusted-ca-bundle\") pod \"d9184e21-8342-4a91-894b-428380062f8f\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") "
Sep 08 18:27:15.704595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704508    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-console-config\") pod \"d9184e21-8342-4a91-894b-428380062f8f\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") "
Sep 08 18:27:15.704595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-oauth-config\") pod \"d9184e21-8342-4a91-894b-428380062f8f\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") "
Sep 08 18:27:15.704865 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704672    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-serving-cert\") pod \"d9184e21-8342-4a91-894b-428380062f8f\" (UID: \"d9184e21-8342-4a91-894b-428380062f8f\") "
Sep 08 18:27:15.704921 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704868    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9184e21-8342-4a91-894b-428380062f8f" (UID: "d9184e21-8342-4a91-894b-428380062f8f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:27:15.704921 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d9184e21-8342-4a91-894b-428380062f8f" (UID: "d9184e21-8342-4a91-894b-428380062f8f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:27:15.705024 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.704938    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9184e21-8342-4a91-894b-428380062f8f" (UID: "d9184e21-8342-4a91-894b-428380062f8f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:27:15.705086 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.705037    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:27:15.705086 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.705056    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-oauth-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:27:15.705086 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.705073    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-trusted-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:27:15.705221 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.705076    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-console-config" (OuterVolumeSpecName: "console-config") pod "d9184e21-8342-4a91-894b-428380062f8f" (UID: "d9184e21-8342-4a91-894b-428380062f8f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:27:15.706824 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.706803    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9184e21-8342-4a91-894b-428380062f8f" (UID: "d9184e21-8342-4a91-894b-428380062f8f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:27:15.706909 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.706894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9184e21-8342-4a91-894b-428380062f8f-kube-api-access-f5rf2" (OuterVolumeSpecName: "kube-api-access-f5rf2") pod "d9184e21-8342-4a91-894b-428380062f8f" (UID: "d9184e21-8342-4a91-894b-428380062f8f"). InnerVolumeSpecName "kube-api-access-f5rf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:27:15.706966 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.706945    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9184e21-8342-4a91-894b-428380062f8f" (UID: "d9184e21-8342-4a91-894b-428380062f8f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:27:15.805522 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.805488    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:27:15.805522 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.805516    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rf2\" (UniqueName: \"kubernetes.io/projected/d9184e21-8342-4a91-894b-428380062f8f-kube-api-access-f5rf2\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:27:15.805522 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.805530    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9184e21-8342-4a91-894b-428380062f8f-console-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:27:15.805747 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:15.805539    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9184e21-8342-4a91-894b-428380062f8f-console-oauth-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:27:16.130974 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.130938    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9184e21-8342-4a91-894b-428380062f8f" containerID="af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b" exitCode=0
Sep 08 18:27:16.131132 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.130985    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fc978bc-65f6p"
Sep 08 18:27:16.131132 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.131015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fc978bc-65f6p" event={"ID":"d9184e21-8342-4a91-894b-428380062f8f","Type":"ContainerDied","Data":"af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b"}
Sep 08 18:27:16.131132 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.131084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fc978bc-65f6p" event={"ID":"d9184e21-8342-4a91-894b-428380062f8f","Type":"ContainerDied","Data":"bef4cbac238815b2317e365b4307e6b24fda3043df20f4d2ffae8fae4f4dff0d"}
Sep 08 18:27:16.131132 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.131103    2580 scope.go:117] "RemoveContainer" containerID="af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b"
Sep 08 18:27:16.139150 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.139131    2580 scope.go:117] "RemoveContainer" containerID="af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b"
Sep 08 18:27:16.139445 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:27:16.139403    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b\": container with ID starting with af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b not found: ID does not exist" containerID="af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b"
Sep 08 18:27:16.139510 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.139446    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b"} err="failed to get container status \"af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b\": rpc error: code = NotFound desc = could not find container \"af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b\": container with ID starting with af1229cb0f6b4125a373bab178754d3f342c3ecff22f6a9d12213f75f4fae64b not found: ID does not exist"
Sep 08 18:27:16.149862 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.149843    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666fc978bc-65f6p"]
Sep 08 18:27:16.154082 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:16.154065    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-666fc978bc-65f6p"]
Sep 08 18:27:18.119875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:18.119844    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9184e21-8342-4a91-894b-428380062f8f" path="/var/lib/kubelet/pods/d9184e21-8342-4a91-894b-428380062f8f/volumes"
Sep 08 18:27:37.345327 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:37.345271    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:27:37.361200 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:37.361172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:27:38.233595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:27:38.233570    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 18:28:38.030666 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:28:38.030641    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 18:30:00.822447 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.822390    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"]
Sep 08 18:30:00.823042 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.822833    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9184e21-8342-4a91-894b-428380062f8f" containerName="console"
Sep 08 18:30:00.823042 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.822849    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9184e21-8342-4a91-894b-428380062f8f" containerName="console"
Sep 08 18:30:00.823042 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.822929    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9184e21-8342-4a91-894b-428380062f8f" containerName="console"
Sep 08 18:30:00.825987 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.825966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:00.829745 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.829720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 18:30:00.829892 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.829867    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 18:30:00.830247 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.830229    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w9cwx\""
Sep 08 18:30:00.830345 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.830331    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 18:30:00.830449 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.830411    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 18:30:00.847844 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.847819    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"]
Sep 08 18:30:00.949587 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.949553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb6b9efa-19f3-42ea-8a09-560d57660a20-manifests\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:00.949770 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.949593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb6b9efa-19f3-42ea-8a09-560d57660a20-webhook-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:00.949770 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.949619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb6b9efa-19f3-42ea-8a09-560d57660a20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:00.949770 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.949702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb6b9efa-19f3-42ea-8a09-560d57660a20-tmp\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:00.949889 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:00.949773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpt52\" (UniqueName: \"kubernetes.io/projected/eb6b9efa-19f3-42ea-8a09-560d57660a20-kube-api-access-lpt52\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.050786 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.050751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpt52\" (UniqueName: \"kubernetes.io/projected/eb6b9efa-19f3-42ea-8a09-560d57660a20-kube-api-access-lpt52\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.050963 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.050792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb6b9efa-19f3-42ea-8a09-560d57660a20-manifests\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.050963 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.050832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb6b9efa-19f3-42ea-8a09-560d57660a20-webhook-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.050963 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.050861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb6b9efa-19f3-42ea-8a09-560d57660a20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.050963 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.050904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb6b9efa-19f3-42ea-8a09-560d57660a20-tmp\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.051258 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.051232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb6b9efa-19f3-42ea-8a09-560d57660a20-manifests\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.053402 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.053372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb6b9efa-19f3-42ea-8a09-560d57660a20-tmp\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.053608 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.053589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb6b9efa-19f3-42ea-8a09-560d57660a20-webhook-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.053667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.053593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb6b9efa-19f3-42ea-8a09-560d57660a20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.059135 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.059113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpt52\" (UniqueName: \"kubernetes.io/projected/eb6b9efa-19f3-42ea-8a09-560d57660a20-kube-api-access-lpt52\") pod \"opendatahub-operator-controller-manager-8d688d88f-g5r25\" (UID: \"eb6b9efa-19f3-42ea-8a09-560d57660a20\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.136506 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.136474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:01.420566 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.420494    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"]
Sep 08 18:30:01.424217 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:30:01.424190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6b9efa_19f3_42ea_8a09_560d57660a20.slice/crio-687472fdeca66d421da4b6435ea7dace9d5ae704ca9014388ec4c047d4d8ff28 WatchSource:0}: Error finding container 687472fdeca66d421da4b6435ea7dace9d5ae704ca9014388ec4c047d4d8ff28: Status 404 returned error can't find the container with id 687472fdeca66d421da4b6435ea7dace9d5ae704ca9014388ec4c047d4d8ff28
Sep 08 18:30:01.426686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.426671    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:30:01.752048 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:01.751954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25" event={"ID":"eb6b9efa-19f3-42ea-8a09-560d57660a20","Type":"ContainerStarted","Data":"687472fdeca66d421da4b6435ea7dace9d5ae704ca9014388ec4c047d4d8ff28"}
Sep 08 18:30:05.774893 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:05.774853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25" event={"ID":"eb6b9efa-19f3-42ea-8a09-560d57660a20","Type":"ContainerStarted","Data":"b37b0c88dd62faf1241c0a08cb97d54385ff4409442f82a0b08e04b61ba41e85"}
Sep 08 18:30:06.781230 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:06.781194    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb6b9efa-19f3-42ea-8a09-560d57660a20" containerID="b37b0c88dd62faf1241c0a08cb97d54385ff4409442f82a0b08e04b61ba41e85" exitCode=0
Sep 08 18:30:06.781606 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:06.781274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25" event={"ID":"eb6b9efa-19f3-42ea-8a09-560d57660a20","Type":"ContainerDied","Data":"b37b0c88dd62faf1241c0a08cb97d54385ff4409442f82a0b08e04b61ba41e85"}
Sep 08 18:30:07.789065 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:07.789029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25" event={"ID":"eb6b9efa-19f3-42ea-8a09-560d57660a20","Type":"ContainerStarted","Data":"30eb70fe3a2623b0f074df61df3ea4371e0b396ed51afae8ff24842b9cb75efe"}
Sep 08 18:30:07.789481 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:07.789164    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:07.813868 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:07.813816    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25" podStartSLOduration=2.767742494 podStartE2EDuration="7.813798504s" totalImagesPullingTime="4.031979026s" totalInitContainerRuntime="1.014076984s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:07.811363588 +0000 UTC m=+390.221673681" watchObservedRunningTime="2026-09-08 18:30:07.813798504 +0000 UTC m=+390.224108585"
Sep 08 18:30:18.795539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:18.795507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-g5r25"
Sep 08 18:30:20.656526 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.656493    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"]
Sep 08 18:30:20.669474 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.669448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.672499 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.672467    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 18:30:20.672653 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.672508    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 18:30:20.673676 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.673656    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:30:20.673780 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.673694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-s8snx\""
Sep 08 18:30:20.673855 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.673818    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 18:30:20.673943 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.673919    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 18:30:20.683042 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.683024    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"]
Sep 08 18:30:20.729447 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.729401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c084bd09-dd62-469d-8738-2ab54fe78e51-cert\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.729610 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.729481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c084bd09-dd62-469d-8738-2ab54fe78e51-metrics-cert\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.729610 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.729512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2jc2\" (UniqueName: \"kubernetes.io/projected/c084bd09-dd62-469d-8738-2ab54fe78e51-kube-api-access-s2jc2\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.729610 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.729591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c084bd09-dd62-469d-8738-2ab54fe78e51-manager-config\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.829983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.829949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c084bd09-dd62-469d-8738-2ab54fe78e51-manager-config\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.830145 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.830008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c084bd09-dd62-469d-8738-2ab54fe78e51-cert\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.830145 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.830033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c084bd09-dd62-469d-8738-2ab54fe78e51-metrics-cert\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.830145 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.830050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2jc2\" (UniqueName: \"kubernetes.io/projected/c084bd09-dd62-469d-8738-2ab54fe78e51-kube-api-access-s2jc2\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.830727 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.830702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c084bd09-dd62-469d-8738-2ab54fe78e51-manager-config\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.832726 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.832700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c084bd09-dd62-469d-8738-2ab54fe78e51-metrics-cert\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.832859 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.832814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c084bd09-dd62-469d-8738-2ab54fe78e51-cert\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.854549 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.854522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2jc2\" (UniqueName: \"kubernetes.io/projected/c084bd09-dd62-469d-8738-2ab54fe78e51-kube-api-access-s2jc2\") pod \"lws-controller-manager-6b68db98f8-hltth\" (UID: \"c084bd09-dd62-469d-8738-2ab54fe78e51\") " pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:20.980804 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:20.980722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:21.262070 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:21.262041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"]
Sep 08 18:30:21.264551 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:30:21.264506    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc084bd09_dd62_469d_8738_2ab54fe78e51.slice/crio-456b797e98632f0006ca3fe4ec8bfe8e628595dfe3b88ea18c0ed81c63da9bb1 WatchSource:0}: Error finding container 456b797e98632f0006ca3fe4ec8bfe8e628595dfe3b88ea18c0ed81c63da9bb1: Status 404 returned error can't find the container with id 456b797e98632f0006ca3fe4ec8bfe8e628595dfe3b88ea18c0ed81c63da9bb1
Sep 08 18:30:21.857087 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:21.857056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth" event={"ID":"c084bd09-dd62-469d-8738-2ab54fe78e51","Type":"ContainerStarted","Data":"456b797e98632f0006ca3fe4ec8bfe8e628595dfe3b88ea18c0ed81c63da9bb1"}
Sep 08 18:30:23.867631 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:23.867599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth" event={"ID":"c084bd09-dd62-469d-8738-2ab54fe78e51","Type":"ContainerStarted","Data":"ae832b493ede7894bac18a06d61af6803eceb73f7ac92dd3e56b5c182e5effb2"}
Sep 08 18:30:23.867977 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:23.867672    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:23.886311 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:23.886259    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth" podStartSLOduration=1.5941444470000001 podStartE2EDuration="3.886238672s" totalImagesPullingTime="2.292094225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:23.884799366 +0000 UTC m=+406.295109472" watchObservedRunningTime="2026-09-08 18:30:23.886238672 +0000 UTC m=+406.296548753"
Sep 08 18:30:34.874946 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:34.874908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6b68db98f8-hltth"
Sep 08 18:30:38.911617 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.911581    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"]
Sep 08 18:30:38.916699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.916676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:38.919606 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.919580    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 18:30:38.919606 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.919581    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 18:30:38.919796 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.919644    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 18:30:38.919796 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.919672    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 18:30:38.919796 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.919711    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 18:30:38.919796 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.919673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8dbrq\""
Sep 08 18:30:38.919948 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.919798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 18:30:38.927792 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.927772    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"]
Sep 08 18:30:38.995784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.995410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjdc\" (UniqueName: \"kubernetes.io/projected/8e9facc1-6925-4fc1-b329-f8253a7a8c09-kube-api-access-gqjdc\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:38.995784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.995501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e9facc1-6925-4fc1-b329-f8253a7a8c09-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:38.995784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.995527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:38.995784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.995576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:38.995784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.995601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:38.995784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.995642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:38.995784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:38.995688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.096713 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.096680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.096713 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.096728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.096946 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.096756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.096946 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.096787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.096946 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.096820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjdc\" (UniqueName: \"kubernetes.io/projected/8e9facc1-6925-4fc1-b329-f8253a7a8c09-kube-api-access-gqjdc\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.097108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.096957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e9facc1-6925-4fc1-b329-f8253a7a8c09-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.097108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.096993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.097725 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.097698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.099311 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.099281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e9facc1-6925-4fc1-b329-f8253a7a8c09-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.099453 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.099347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.099534 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.099488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.099620 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.099602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.110532 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.110501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjdc\" (UniqueName: \"kubernetes.io/projected/8e9facc1-6925-4fc1-b329-f8253a7a8c09-kube-api-access-gqjdc\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.110856 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.110838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e9facc1-6925-4fc1-b329-f8253a7a8c09-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kw8r7\" (UID: \"8e9facc1-6925-4fc1-b329-f8253a7a8c09\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.227591 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.227500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:39.537707 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.537627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"]
Sep 08 18:30:39.540616 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:30:39.540586    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9facc1_6925_4fc1_b329_f8253a7a8c09.slice/crio-62dc01c19f70aec693bb11cef8e499272498e22a8feb53fd055ce59834eca88d WatchSource:0}: Error finding container 62dc01c19f70aec693bb11cef8e499272498e22a8feb53fd055ce59834eca88d: Status 404 returned error can't find the container with id 62dc01c19f70aec693bb11cef8e499272498e22a8feb53fd055ce59834eca88d
Sep 08 18:30:39.943419 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:39.943380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7" event={"ID":"8e9facc1-6925-4fc1-b329-f8253a7a8c09","Type":"ContainerStarted","Data":"62dc01c19f70aec693bb11cef8e499272498e22a8feb53fd055ce59834eca88d"}
Sep 08 18:30:42.146661 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:42.146615    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 08 18:30:42.146913 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:42.146690    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 08 18:30:42.959292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:42.959256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7" event={"ID":"8e9facc1-6925-4fc1-b329-f8253a7a8c09","Type":"ContainerStarted","Data":"03b4c7d5659dae5ebd8d7f370313e72241baf4719cd40e1abc9972421b36a511"}
Sep 08 18:30:42.959464 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:42.959377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:30:42.983478 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:42.983413    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7" podStartSLOduration=2.379664766 podStartE2EDuration="4.983397767s" totalImagesPullingTime="2.603733001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:42.982084294 +0000 UTC m=+425.392394376" watchObservedRunningTime="2026-09-08 18:30:42.983397767 +0000 UTC m=+425.393707848"
Sep 08 18:30:43.964838 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:30:43.964806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kw8r7"
Sep 08 18:31:55.337343 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.337304    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"]
Sep 08 18:31:55.340990 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.340968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"
Sep 08 18:31:55.343807 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.343782    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 18:31:55.343929 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.343839    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bsl9r\""
Sep 08 18:31:55.343929 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.343871    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 18:31:55.343929 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.343839    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 08 18:31:55.353107 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.353072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"]
Sep 08 18:31:55.478515 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.478459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4q9g\" (UniqueName: \"kubernetes.io/projected/73cca14f-0dcc-4b7c-a2a5-acf5144bded6-kube-api-access-j4q9g\") pod \"dns-operator-controller-manager-7b84cd7c9c-lxbp4\" (UID: \"73cca14f-0dcc-4b7c-a2a5-acf5144bded6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"
Sep 08 18:31:55.579633 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.579597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4q9g\" (UniqueName: \"kubernetes.io/projected/73cca14f-0dcc-4b7c-a2a5-acf5144bded6-kube-api-access-j4q9g\") pod \"dns-operator-controller-manager-7b84cd7c9c-lxbp4\" (UID: \"73cca14f-0dcc-4b7c-a2a5-acf5144bded6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"
Sep 08 18:31:55.591753 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.591686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4q9g\" (UniqueName: \"kubernetes.io/projected/73cca14f-0dcc-4b7c-a2a5-acf5144bded6-kube-api-access-j4q9g\") pod \"dns-operator-controller-manager-7b84cd7c9c-lxbp4\" (UID: \"73cca14f-0dcc-4b7c-a2a5-acf5144bded6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"
Sep 08 18:31:55.650680 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.650656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"
Sep 08 18:31:55.938951 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:55.936834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"]
Sep 08 18:31:55.949767 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:31:55.949723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cca14f_0dcc_4b7c_a2a5_acf5144bded6.slice/crio-24a6cecb973d5145f8fc4b0974ad4478ce2a4002082fbdf3342fe7a98aefc66a WatchSource:0}: Error finding container 24a6cecb973d5145f8fc4b0974ad4478ce2a4002082fbdf3342fe7a98aefc66a: Status 404 returned error can't find the container with id 24a6cecb973d5145f8fc4b0974ad4478ce2a4002082fbdf3342fe7a98aefc66a
Sep 08 18:31:56.272572 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:56.272495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4" event={"ID":"73cca14f-0dcc-4b7c-a2a5-acf5144bded6","Type":"ContainerStarted","Data":"24a6cecb973d5145f8fc4b0974ad4478ce2a4002082fbdf3342fe7a98aefc66a"}
Sep 08 18:31:59.293185 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:59.293150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4" event={"ID":"73cca14f-0dcc-4b7c-a2a5-acf5144bded6","Type":"ContainerStarted","Data":"c25b712b9351566d2ea449794bb479a6094346a428cb0cde74608287cbd47d04"}
Sep 08 18:31:59.293661 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:59.293300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"
Sep 08 18:31:59.311767 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:31:59.311714    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4" podStartSLOduration=1.52693996 podStartE2EDuration="4.311697047s" totalImagesPullingTime="2.784757087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:31:59.309985436 +0000 UTC m=+501.720295547" watchObservedRunningTime="2026-09-08 18:31:59.311697047 +0000 UTC m=+501.722007128"
Sep 08 18:32:07.110716 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.110685    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"]
Sep 08 18:32:07.113965 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.113945    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"
Sep 08 18:32:07.116969 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.116946    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9ld8f\""
Sep 08 18:32:07.127241 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.127218    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"]
Sep 08 18:32:07.163665 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.163637    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"]
Sep 08 18:32:07.163916 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:32:07.163897    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-n8msr], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-n8msr]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr" podUID="4b3d437a-7ea2-4ba8-b2b3-33908f782363"
Sep 08 18:32:07.172157 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.172129    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"]
Sep 08 18:32:07.194019 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.193992    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"]
Sep 08 18:32:07.197487 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.197467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:07.220194 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.220158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"]
Sep 08 18:32:07.278981 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.278951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jffjs\" (UniqueName: \"kubernetes.io/projected/4d13dee0-de08-472a-a28f-c8ffb2b51421-kube-api-access-jffjs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-84vrd\" (UID: \"4d13dee0-de08-472a-a28f-c8ffb2b51421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:07.279138 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.279018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d13dee0-de08-472a-a28f-c8ffb2b51421-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-84vrd\" (UID: \"4d13dee0-de08-472a-a28f-c8ffb2b51421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:07.328591 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.328559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"
Sep 08 18:32:07.331084 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:32:07.331056    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-xn5zr\" is forbidden: User \"system:node:ip-10-0-141-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-6.ec2.internal' and this object" podUID="4b3d437a-7ea2-4ba8-b2b3-33908f782363" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"
Sep 08 18:32:07.335063 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.335044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"
Sep 08 18:32:07.337613 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:32:07.337590    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-xn5zr\" is forbidden: User \"system:node:ip-10-0-141-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-6.ec2.internal' and this object" podUID="4b3d437a-7ea2-4ba8-b2b3-33908f782363" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"
Sep 08 18:32:07.380419 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.380392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d13dee0-de08-472a-a28f-c8ffb2b51421-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-84vrd\" (UID: \"4d13dee0-de08-472a-a28f-c8ffb2b51421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:07.380577 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.380487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jffjs\" (UniqueName: \"kubernetes.io/projected/4d13dee0-de08-472a-a28f-c8ffb2b51421-kube-api-access-jffjs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-84vrd\" (UID: \"4d13dee0-de08-472a-a28f-c8ffb2b51421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:07.380809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.380786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d13dee0-de08-472a-a28f-c8ffb2b51421-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-84vrd\" (UID: \"4d13dee0-de08-472a-a28f-c8ffb2b51421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:07.390701 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.390680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jffjs\" (UniqueName: \"kubernetes.io/projected/4d13dee0-de08-472a-a28f-c8ffb2b51421-kube-api-access-jffjs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-84vrd\" (UID: \"4d13dee0-de08-472a-a28f-c8ffb2b51421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:07.508358 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.508325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:07.806257 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:07.806235    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"]
Sep 08 18:32:07.807877 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:32:07.807852    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d13dee0_de08_472a_a28f_c8ffb2b51421.slice/crio-cb189a69059a68b4c2c07a0d5b010ced8ea11805272c21a6996b6b1acfe8ac89 WatchSource:0}: Error finding container cb189a69059a68b4c2c07a0d5b010ced8ea11805272c21a6996b6b1acfe8ac89: Status 404 returned error can't find the container with id cb189a69059a68b4c2c07a0d5b010ced8ea11805272c21a6996b6b1acfe8ac89
Sep 08 18:32:08.118555 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:32:08.118525    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-xn5zr\" is forbidden: User \"system:node:ip-10-0-141-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-6.ec2.internal' and this object" podUID="4b3d437a-7ea2-4ba8-b2b3-33908f782363" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"
Sep 08 18:32:08.119865 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:08.119838    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b3d437a-7ea2-4ba8-b2b3-33908f782363" path="/var/lib/kubelet/pods/4b3d437a-7ea2-4ba8-b2b3-33908f782363/volumes"
Sep 08 18:32:08.335012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:08.334982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"
Sep 08 18:32:08.335012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:08.334996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd" event={"ID":"4d13dee0-de08-472a-a28f-c8ffb2b51421","Type":"ContainerStarted","Data":"cb189a69059a68b4c2c07a0d5b010ced8ea11805272c21a6996b6b1acfe8ac89"}
Sep 08 18:32:08.339791 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:32:08.339758    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-xn5zr\" is forbidden: User \"system:node:ip-10-0-141-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-6.ec2.internal' and this object" podUID="4b3d437a-7ea2-4ba8-b2b3-33908f782363" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xn5zr"
Sep 08 18:32:10.300793 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:10.300669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lxbp4"
Sep 08 18:32:12.357044 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:12.357013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd" event={"ID":"4d13dee0-de08-472a-a28f-c8ffb2b51421","Type":"ContainerStarted","Data":"658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a"}
Sep 08 18:32:12.357501 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:12.357134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:32:12.382115 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:12.382068    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd" podStartSLOduration=1.742533124 podStartE2EDuration="5.382050639s" totalImagesPullingTime="3.639517515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:32:12.379906148 +0000 UTC m=+514.790216232" watchObservedRunningTime="2026-09-08 18:32:12.382050639 +0000 UTC m=+514.792360719"
Sep 08 18:32:23.364168 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:32:23.364138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:33:30.222444 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.222403    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-587bbc7cb-7xk2j"]
Sep 08 18:33:30.225918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.225897    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.228794 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.228768    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 18:33:30.228979 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.228836    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 18:33:30.228979 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.228863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r7hq9\""
Sep 08 18:33:30.228979 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.228878    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 18:33:30.238215 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.238194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-7xk2j"]
Sep 08 18:33:30.298410 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.298370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-metrics-certs\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.298607 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.298516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdxd\" (UniqueName: \"kubernetes.io/projected/49f16ab7-a217-4076-8b94-b8132cc7a89c-kube-api-access-4jdxd\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.298607 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.298551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.399784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.399750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-metrics-certs\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.399974 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.399817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdxd\" (UniqueName: \"kubernetes.io/projected/49f16ab7-a217-4076-8b94-b8132cc7a89c-kube-api-access-4jdxd\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.399974 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.399837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.399974 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:33:30.399936    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 08 18:33:30.400148 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:33:30.400001    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert podName:49f16ab7-a217-4076-8b94-b8132cc7a89c nodeName:}" failed. No retries permitted until 2026-09-08 18:33:30.899984643 +0000 UTC m=+593.310294701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert") pod "maas-controller-587bbc7cb-7xk2j" (UID: "49f16ab7-a217-4076-8b94-b8132cc7a89c") : secret "maas-controller-webhook-cert" not found
Sep 08 18:33:30.402281 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.402258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-metrics-certs\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.414198 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.414165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdxd\" (UniqueName: \"kubernetes.io/projected/49f16ab7-a217-4076-8b94-b8132cc7a89c-kube-api-access-4jdxd\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.421076 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.421055    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-l6c7j"]
Sep 08 18:33:30.424615 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.424599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.438740 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.434826    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-l6c7j"]
Sep 08 18:33:30.501034 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.500943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89mm\" (UniqueName: \"kubernetes.io/projected/102a765e-9699-40c7-a763-83d46e5b89ff-kube-api-access-s89mm\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.501034 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.500994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-metrics-certs\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.501034 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.501028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-webhook-cert\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.602084 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.602051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89mm\" (UniqueName: \"kubernetes.io/projected/102a765e-9699-40c7-a763-83d46e5b89ff-kube-api-access-s89mm\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.602255 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.602097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-metrics-certs\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.602300 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.602280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-webhook-cert\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.604709 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.604680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-webhook-cert\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.604830 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.604810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-metrics-certs\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.612075 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.612049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89mm\" (UniqueName: \"kubernetes.io/projected/102a765e-9699-40c7-a763-83d46e5b89ff-kube-api-access-s89mm\") pod \"maas-controller-865fc4c5d4-l6c7j\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") " pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.735918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.735881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:30.905305 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.905270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:30.912634 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:30.912606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert\") pod \"maas-controller-587bbc7cb-7xk2j\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") " pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:31.006523 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:31.006488    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-l6c7j"]
Sep 08 18:33:31.010312 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:33:31.010287    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102a765e_9699_40c7_a763_83d46e5b89ff.slice/crio-9abde93a5e12046b8c2cac6f5b5b59f80129ee7f606c34bffa7702647ab2b0e7 WatchSource:0}: Error finding container 9abde93a5e12046b8c2cac6f5b5b59f80129ee7f606c34bffa7702647ab2b0e7: Status 404 returned error can't find the container with id 9abde93a5e12046b8c2cac6f5b5b59f80129ee7f606c34bffa7702647ab2b0e7
Sep 08 18:33:31.136279 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:31.136239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:31.409989 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:31.409916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-7xk2j"]
Sep 08 18:33:31.413383 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:33:31.413348    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f16ab7_a217_4076_8b94_b8132cc7a89c.slice/crio-ea4c7e946be043ba61a8d43834c6d00aa8b486a6aa5da24fb01d77f0c1a8cea5 WatchSource:0}: Error finding container ea4c7e946be043ba61a8d43834c6d00aa8b486a6aa5da24fb01d77f0c1a8cea5: Status 404 returned error can't find the container with id ea4c7e946be043ba61a8d43834c6d00aa8b486a6aa5da24fb01d77f0c1a8cea5
Sep 08 18:33:31.717144 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:31.717057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587bbc7cb-7xk2j" event={"ID":"49f16ab7-a217-4076-8b94-b8132cc7a89c","Type":"ContainerStarted","Data":"ea4c7e946be043ba61a8d43834c6d00aa8b486a6aa5da24fb01d77f0c1a8cea5"}
Sep 08 18:33:31.718232 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:31.718209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j" event={"ID":"102a765e-9699-40c7-a763-83d46e5b89ff","Type":"ContainerStarted","Data":"9abde93a5e12046b8c2cac6f5b5b59f80129ee7f606c34bffa7702647ab2b0e7"}
Sep 08 18:33:33.730172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:33.730126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j" event={"ID":"102a765e-9699-40c7-a763-83d46e5b89ff","Type":"ContainerStarted","Data":"1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850"}
Sep 08 18:33:33.730671 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:33.730234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:33.731786 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:33.731763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587bbc7cb-7xk2j" event={"ID":"49f16ab7-a217-4076-8b94-b8132cc7a89c","Type":"ContainerStarted","Data":"3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442"}
Sep 08 18:33:33.731898 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:33.731803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:33.751762 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:33.751707    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j" podStartSLOduration=1.674144012 podStartE2EDuration="3.751690063s" totalImagesPullingTime="2.077546051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:33:33.74889064 +0000 UTC m=+596.159200722" watchObservedRunningTime="2026-09-08 18:33:33.751690063 +0000 UTC m=+596.162000144"
Sep 08 18:33:33.773578 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:33.773524    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-587bbc7cb-7xk2j" podStartSLOduration=2.079363349 podStartE2EDuration="3.77350443s" totalImagesPullingTime="1.694141081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:33:33.771725268 +0000 UTC m=+596.182035380" watchObservedRunningTime="2026-09-08 18:33:33.77350443 +0000 UTC m=+596.183814510"
Sep 08 18:33:44.739147 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:44.739113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:33:44.739543 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:44.739285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:44.805117 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:44.805086    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-7xk2j"]
Sep 08 18:33:44.805379 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:44.805356    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-587bbc7cb-7xk2j" podUID="49f16ab7-a217-4076-8b94-b8132cc7a89c" containerName="manager" containerID="cri-o://3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442" gracePeriod=30
Sep 08 18:33:45.045287 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.045263    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:45.134058 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.134024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jdxd\" (UniqueName: \"kubernetes.io/projected/49f16ab7-a217-4076-8b94-b8132cc7a89c-kube-api-access-4jdxd\") pod \"49f16ab7-a217-4076-8b94-b8132cc7a89c\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") "
Sep 08 18:33:45.134058 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.134065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert\") pod \"49f16ab7-a217-4076-8b94-b8132cc7a89c\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") "
Sep 08 18:33:45.134313 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.134086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-metrics-certs\") pod \"49f16ab7-a217-4076-8b94-b8132cc7a89c\" (UID: \"49f16ab7-a217-4076-8b94-b8132cc7a89c\") "
Sep 08 18:33:45.136526 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.136486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "49f16ab7-a217-4076-8b94-b8132cc7a89c" (UID: "49f16ab7-a217-4076-8b94-b8132cc7a89c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:33:45.136526 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.136510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "49f16ab7-a217-4076-8b94-b8132cc7a89c" (UID: "49f16ab7-a217-4076-8b94-b8132cc7a89c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:33:45.136526 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.136487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f16ab7-a217-4076-8b94-b8132cc7a89c-kube-api-access-4jdxd" (OuterVolumeSpecName: "kube-api-access-4jdxd") pod "49f16ab7-a217-4076-8b94-b8132cc7a89c" (UID: "49f16ab7-a217-4076-8b94-b8132cc7a89c"). InnerVolumeSpecName "kube-api-access-4jdxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:33:45.235568 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.235528    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdxd\" (UniqueName: \"kubernetes.io/projected/49f16ab7-a217-4076-8b94-b8132cc7a89c-kube-api-access-4jdxd\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:33:45.235568 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.235561    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-webhook-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:33:45.235568 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.235571    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f16ab7-a217-4076-8b94-b8132cc7a89c-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:33:45.789257 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.789221    2580 generic.go:356] "Generic (PLEG): container finished" podID="49f16ab7-a217-4076-8b94-b8132cc7a89c" containerID="3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442" exitCode=0
Sep 08 18:33:45.789694 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.789272    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587bbc7cb-7xk2j"
Sep 08 18:33:45.789694 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.789307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587bbc7cb-7xk2j" event={"ID":"49f16ab7-a217-4076-8b94-b8132cc7a89c","Type":"ContainerDied","Data":"3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442"}
Sep 08 18:33:45.789694 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.789364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587bbc7cb-7xk2j" event={"ID":"49f16ab7-a217-4076-8b94-b8132cc7a89c","Type":"ContainerDied","Data":"ea4c7e946be043ba61a8d43834c6d00aa8b486a6aa5da24fb01d77f0c1a8cea5"}
Sep 08 18:33:45.789694 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.789387    2580 scope.go:117] "RemoveContainer" containerID="3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442"
Sep 08 18:33:45.798442 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.798406    2580 scope.go:117] "RemoveContainer" containerID="3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442"
Sep 08 18:33:45.798696 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:33:45.798678    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442\": container with ID starting with 3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442 not found: ID does not exist" containerID="3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442"
Sep 08 18:33:45.798740 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.798704    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442"} err="failed to get container status \"3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442\": rpc error: code = NotFound desc = could not find container \"3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442\": container with ID starting with 3dab6e1310db6251bb8f94ebba07dcbf3b3ef9e2942e89cad8e8ea6a37d57442 not found: ID does not exist"
Sep 08 18:33:45.810915 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.810896    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-7xk2j"]
Sep 08 18:33:45.815244 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:45.815223    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-7xk2j"]
Sep 08 18:33:46.121152 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:33:46.121119    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49f16ab7-a217-4076-8b94-b8132cc7a89c" path="/var/lib/kubelet/pods/49f16ab7-a217-4076-8b94-b8132cc7a89c/volumes"
Sep 08 18:34:06.411744 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.411672    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"]
Sep 08 18:34:06.412164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.412018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f16ab7-a217-4076-8b94-b8132cc7a89c" containerName="manager"
Sep 08 18:34:06.412164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.412029    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f16ab7-a217-4076-8b94-b8132cc7a89c" containerName="manager"
Sep 08 18:34:06.412164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.412086    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f16ab7-a217-4076-8b94-b8132cc7a89c" containerName="manager"
Sep 08 18:34:06.415102 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.415084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.417934 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.417912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:34:06.418030 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.417992    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:34:06.419175 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.419157    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:34:06.419256 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.419236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 18:34:06.419256 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.419251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wvhnh\""
Sep 08 18:34:06.428697 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.428674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"]
Sep 08 18:34:06.515319 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.515276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-metrics-certs\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.515499 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.515325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af19ccf-acde-4326-8146-513f4df9dcf9-openshift-service-ca\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.515499 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.515389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7b7\" (UniqueName: \"kubernetes.io/projected/1af19ccf-acde-4326-8146-513f4df9dcf9-kube-api-access-7d7b7\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.515499 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.515463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-maas-api-tls\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.616701 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.616656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-metrics-certs\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.616898 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.616710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af19ccf-acde-4326-8146-513f4df9dcf9-openshift-service-ca\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.616898 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.616754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7b7\" (UniqueName: \"kubernetes.io/projected/1af19ccf-acde-4326-8146-513f4df9dcf9-kube-api-access-7d7b7\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.616898 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.616817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-maas-api-tls\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.617684 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.617660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af19ccf-acde-4326-8146-513f4df9dcf9-openshift-service-ca\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.619322 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.619298    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-metrics-certs\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.619409 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.619334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-maas-api-tls\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.628131 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.628112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7b7\" (UniqueName: \"kubernetes.io/projected/1af19ccf-acde-4326-8146-513f4df9dcf9-kube-api-access-7d7b7\") pod \"maas-api-dd87ddf4c-tqf5s\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:06.725823 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:06.725734    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:07.010615 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:07.010590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"]
Sep 08 18:34:07.012985 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:34:07.012953    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af19ccf_acde_4326_8146_513f4df9dcf9.slice/crio-925f33c7983e4e6a509eab4dfa69b94d1367c9b23e8710f063241a7d0dda459a WatchSource:0}: Error finding container 925f33c7983e4e6a509eab4dfa69b94d1367c9b23e8710f063241a7d0dda459a: Status 404 returned error can't find the container with id 925f33c7983e4e6a509eab4dfa69b94d1367c9b23e8710f063241a7d0dda459a
Sep 08 18:34:07.887220 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:07.887177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s" event={"ID":"1af19ccf-acde-4326-8146-513f4df9dcf9","Type":"ContainerStarted","Data":"925f33c7983e4e6a509eab4dfa69b94d1367c9b23e8710f063241a7d0dda459a"}
Sep 08 18:34:09.900037 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:09.900003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s" event={"ID":"1af19ccf-acde-4326-8146-513f4df9dcf9","Type":"ContainerStarted","Data":"2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3"}
Sep 08 18:34:09.900496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:09.900078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:09.918825 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:09.918773    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s" podStartSLOduration=1.846698196 podStartE2EDuration="3.918755405s" totalImagesPullingTime="2.072057209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:09.916270707 +0000 UTC m=+632.326580780" watchObservedRunningTime="2026-09-08 18:34:09.918755405 +0000 UTC m=+632.329065486"
Sep 08 18:34:15.910552 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:15.910515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:34:21.939779 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:21.939742    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"]
Sep 08 18:34:21.943591 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:21.943571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:21.947717 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:21.947691    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2gzvd\""
Sep 08 18:34:21.947717 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:21.947703    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 18:34:21.947900 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:21.947765    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 18:34:21.947900 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:21.947817    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 18:34:21.952541 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:21.952033    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"]
Sep 08 18:34:22.057346 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.057308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.057539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.057366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.057539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.057386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.057539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.057456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c2jn\" (UniqueName: \"kubernetes.io/projected/1b642420-8989-4339-a3cf-e427ee2894c6-kube-api-access-5c2jn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.057539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.057487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b642420-8989-4339-a3cf-e427ee2894c6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.057539 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.057538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.158531 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.158502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.158531 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.158537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.158764 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.158558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c2jn\" (UniqueName: \"kubernetes.io/projected/1b642420-8989-4339-a3cf-e427ee2894c6-kube-api-access-5c2jn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.158764 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.158576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b642420-8989-4339-a3cf-e427ee2894c6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.158764 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.158632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.158764 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.158752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.158975 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.158932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.159223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.159185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.159223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.159215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.161242 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.161212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b642420-8989-4339-a3cf-e427ee2894c6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.161469 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.161453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b642420-8989-4339-a3cf-e427ee2894c6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.167613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.167592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c2jn\" (UniqueName: \"kubernetes.io/projected/1b642420-8989-4339-a3cf-e427ee2894c6-kube-api-access-5c2jn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq\" (UID: \"1b642420-8989-4339-a3cf-e427ee2894c6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.256556 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.256468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:22.554547 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:34:22.554512    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b642420_8989_4339_a3cf_e427ee2894c6.slice/crio-c8d587607df9853a4d7a604405bd9d27568d8385724ab3568bea49ef2d5a181e WatchSource:0}: Error finding container c8d587607df9853a4d7a604405bd9d27568d8385724ab3568bea49ef2d5a181e: Status 404 returned error can't find the container with id c8d587607df9853a4d7a604405bd9d27568d8385724ab3568bea49ef2d5a181e
Sep 08 18:34:22.558163 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.558143    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"]
Sep 08 18:34:22.964208 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:22.964166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq" event={"ID":"1b642420-8989-4339-a3cf-e427ee2894c6","Type":"ContainerStarted","Data":"c8d587607df9853a4d7a604405bd9d27568d8385724ab3568bea49ef2d5a181e"}
Sep 08 18:34:23.970952 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:23.970863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq" event={"ID":"1b642420-8989-4339-a3cf-e427ee2894c6","Type":"ContainerStarted","Data":"ea2e59bfd672061ea5ef8a3ae677d9cfacdb8bd5182dea846d38acc0308e6508"}
Sep 08 18:34:23.990871 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:23.990821    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq" podStartSLOduration=1.926033396 podStartE2EDuration="2.990805324s" totalImagesPullingTime="1.064771928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:23.989052366 +0000 UTC m=+646.399362471" watchObservedRunningTime="2026-09-08 18:34:23.990805324 +0000 UTC m=+646.401115405"
Sep 08 18:34:32.256621 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:32.256584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:32.257076 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:32.256646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:32.269382 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:32.269350    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:33.023765 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:33.023738    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq"
Sep 08 18:34:35.113514 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.113476    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"]
Sep 08 18:34:35.116974 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.116947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.119713 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.119677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 08 18:34:35.127129 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.127103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"]
Sep 08 18:34:35.288092 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.288053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.288249 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.288113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.288249 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.288156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a95e427-7b33-4129-a07c-04d930a44236-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.288249 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.288192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xb5d\" (UniqueName: \"kubernetes.io/projected/0a95e427-7b33-4129-a07c-04d930a44236-kube-api-access-2xb5d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.288249 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.288239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.288385 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.288293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.389879 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.389839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.390059 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.389915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.390059 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.389954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a95e427-7b33-4129-a07c-04d930a44236-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.390059 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.389995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xb5d\" (UniqueName: \"kubernetes.io/projected/0a95e427-7b33-4129-a07c-04d930a44236-kube-api-access-2xb5d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.390226 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.390059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.390226 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.390093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.390409 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.390377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.390576 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.390460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.390576 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.390519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.392894 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.392871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a95e427-7b33-4129-a07c-04d930a44236-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.393166 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.393017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a95e427-7b33-4129-a07c-04d930a44236-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.398070 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.398050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xb5d\" (UniqueName: \"kubernetes.io/projected/0a95e427-7b33-4129-a07c-04d930a44236-kube-api-access-2xb5d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjqjt\" (UID: \"0a95e427-7b33-4129-a07c-04d930a44236\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.429155 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.429116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:35.705008 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:35.704980    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"]
Sep 08 18:34:35.707398 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:34:35.707369    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a95e427_7b33_4129_a07c_04d930a44236.slice/crio-a3905efff7c642d5fead50f5564fbe3842e6aeb7ce16e9fa2434a42bd240a2ab WatchSource:0}: Error finding container a3905efff7c642d5fead50f5564fbe3842e6aeb7ce16e9fa2434a42bd240a2ab: Status 404 returned error can't find the container with id a3905efff7c642d5fead50f5564fbe3842e6aeb7ce16e9fa2434a42bd240a2ab
Sep 08 18:34:36.029095 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:36.029006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt" event={"ID":"0a95e427-7b33-4129-a07c-04d930a44236","Type":"ContainerStarted","Data":"dd226ea4b455f323617cab6362131b83a17c58c3abf735138f8ea803e1fedd16"}
Sep 08 18:34:36.029095 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:36.029061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt" event={"ID":"0a95e427-7b33-4129-a07c-04d930a44236","Type":"ContainerStarted","Data":"a3905efff7c642d5fead50f5564fbe3842e6aeb7ce16e9fa2434a42bd240a2ab"}
Sep 08 18:34:36.048355 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:36.048308    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt" podStartSLOduration=0.877055864 podStartE2EDuration="1.048293097s" totalImagesPullingTime="171.237233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:36.045829505 +0000 UTC m=+658.456139580" watchObservedRunningTime="2026-09-08 18:34:36.048293097 +0000 UTC m=+658.458603178"
Sep 08 18:34:45.429782 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:45.429745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:45.430239 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:45.429806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:45.442087 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:45.442061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:34:46.087501 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:34:46.087470    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjqjt"
Sep 08 18:35:19.910689 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:19.910651    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"]
Sep 08 18:35:19.914925 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:19.914896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:19.921991 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:19.921963    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 08 18:35:19.926222 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:19.926162    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"]
Sep 08 18:35:20.093924 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.093888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.094112 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.093931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.094112 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.093956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnb2q\" (UniqueName: \"kubernetes.io/projected/dd4a4e56-f6ab-4395-b3e3-31248364a19c-kube-api-access-pnb2q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.094112 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.094039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4a4e56-f6ab-4395-b3e3-31248364a19c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.094112 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.094086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.094260 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.094136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195246 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnb2q\" (UniqueName: \"kubernetes.io/projected/dd4a4e56-f6ab-4395-b3e3-31248364a19c-kube-api-access-pnb2q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195246 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4a4e56-f6ab-4395-b3e3-31248364a19c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195701 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195701 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.195804 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.195727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.197861 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.197834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4a4e56-f6ab-4395-b3e3-31248364a19c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.198136 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.198115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4a4e56-f6ab-4395-b3e3-31248364a19c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.204192 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.204169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnb2q\" (UniqueName: \"kubernetes.io/projected/dd4a4e56-f6ab-4395-b3e3-31248364a19c-kube-api-access-pnb2q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw\" (UID: \"dd4a4e56-f6ab-4395-b3e3-31248364a19c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.236415 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.236386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:20.507546 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.507507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"]
Sep 08 18:35:20.510135 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:35:20.510104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4a4e56_f6ab_4395_b3e3_31248364a19c.slice/crio-1444c2bd707ae11d139caccd6003488178e889660211aad54306d2e85fd7d1cb WatchSource:0}: Error finding container 1444c2bd707ae11d139caccd6003488178e889660211aad54306d2e85fd7d1cb: Status 404 returned error can't find the container with id 1444c2bd707ae11d139caccd6003488178e889660211aad54306d2e85fd7d1cb
Sep 08 18:35:20.511499 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:20.511484    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:35:21.238201 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:21.238162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw" event={"ID":"dd4a4e56-f6ab-4395-b3e3-31248364a19c","Type":"ContainerStarted","Data":"709fef684a9e9cb2efa08b09d8611d701d8f4b845eb74f6c5c4b6822c4414eab"}
Sep 08 18:35:21.238613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:21.238211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw" event={"ID":"dd4a4e56-f6ab-4395-b3e3-31248364a19c","Type":"ContainerStarted","Data":"1444c2bd707ae11d139caccd6003488178e889660211aad54306d2e85fd7d1cb"}
Sep 08 18:35:21.256579 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:21.256530    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw" podStartSLOduration=2.02824601 podStartE2EDuration="2.256514321s" totalImagesPullingTime="228.268311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:21.254527481 +0000 UTC m=+703.664837559" watchObservedRunningTime="2026-09-08 18:35:21.256514321 +0000 UTC m=+703.666824402"
Sep 08 18:35:30.237098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:30.237003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:30.237098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:30.237077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:30.250293 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:30.250267    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:35:30.292140 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:35:30.292117    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw"
Sep 08 18:36:53.616517 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.616481    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"]
Sep 08 18:36:53.620226 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.620204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.623068 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.623049    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1d8cb1-serving-cert\""
Sep 08 18:36:53.630851 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.630828    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"]
Sep 08 18:36:53.679417 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.679381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b0816e-9492-4fcb-9f34-11fff090173c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.679618 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.679493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-metrics-certs\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.679618 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.679597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcd5b\" (UniqueName: \"kubernetes.io/projected/46b0816e-9492-4fcb-9f34-11fff090173c-kube-api-access-vcd5b\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.679731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.679671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.780740 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.780696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b0816e-9492-4fcb-9f34-11fff090173c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.780928 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.780763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-metrics-certs\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.780928 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.780883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcd5b\" (UniqueName: \"kubernetes.io/projected/46b0816e-9492-4fcb-9f34-11fff090173c-kube-api-access-vcd5b\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.780928 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.780908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.781103 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:36:53.781047    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-serving-cert: secret "maas-api-e2e-inf-a-1d8cb1-serving-cert" not found
Sep 08 18:36:53.781159 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:36:53.781128    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls podName:46b0816e-9492-4fcb-9f34-11fff090173c nodeName:}" failed. No retries permitted until 2026-09-08 18:36:54.281105771 +0000 UTC m=+796.691415846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls") pod "maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z" (UID: "46b0816e-9492-4fcb-9f34-11fff090173c") : secret "maas-api-e2e-inf-a-1d8cb1-serving-cert" not found
Sep 08 18:36:53.781362 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.781340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b0816e-9492-4fcb-9f34-11fff090173c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.783533 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.783508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-metrics-certs\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:53.789483 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:53.789454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcd5b\" (UniqueName: \"kubernetes.io/projected/46b0816e-9492-4fcb-9f34-11fff090173c-kube-api-access-vcd5b\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:54.286588 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:54.286549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:54.289156 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:54.289134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:54.532149 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:54.532113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:54.809235 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:54.809213    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"]
Sep 08 18:36:54.811311 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:36:54.811270    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b0816e_9492_4fcb_9f34_11fff090173c.slice/crio-0e14fcf8723a05f13f0cc7fe1328ab0798669a9ebb7c1184c6259a0d10b2fb5d WatchSource:0}: Error finding container 0e14fcf8723a05f13f0cc7fe1328ab0798669a9ebb7c1184c6259a0d10b2fb5d: Status 404 returned error can't find the container with id 0e14fcf8723a05f13f0cc7fe1328ab0798669a9ebb7c1184c6259a0d10b2fb5d
Sep 08 18:36:55.684905 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:55.684871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z" event={"ID":"46b0816e-9492-4fcb-9f34-11fff090173c","Type":"ContainerStarted","Data":"380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2"}
Sep 08 18:36:55.685101 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:55.684922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z" event={"ID":"46b0816e-9492-4fcb-9f34-11fff090173c","Type":"ContainerStarted","Data":"0e14fcf8723a05f13f0cc7fe1328ab0798669a9ebb7c1184c6259a0d10b2fb5d"}
Sep 08 18:36:55.685101 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:55.684958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:36:55.704115 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:36:55.704065    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z" podStartSLOduration=2.247775139 podStartE2EDuration="2.704048588s" totalImagesPullingTime="456.273449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:55.701921114 +0000 UTC m=+798.112231208" watchObservedRunningTime="2026-09-08 18:36:55.704048588 +0000 UTC m=+798.114358669"
Sep 08 18:37:00.958173 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:00.958085    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"]
Sep 08 18:37:00.961807 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:00.961774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:00.966755 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:00.966729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-abd9a747-serving-cert\""
Sep 08 18:37:00.979960 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:00.979933    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"]
Sep 08 18:37:01.051888 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.051851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-metrics-certs\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.052069 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.051940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.052069 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.051968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lct4d\" (UniqueName: \"kubernetes.io/projected/62ebe370-9531-4a0c-971b-0fa3b75fd468-kube-api-access-lct4d\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.052069 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.052014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ebe370-9531-4a0c-971b-0fa3b75fd468-openshift-service-ca\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.152915 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.152883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.153066 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.152924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lct4d\" (UniqueName: \"kubernetes.io/projected/62ebe370-9531-4a0c-971b-0fa3b75fd468-kube-api-access-lct4d\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.153066 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.152976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ebe370-9531-4a0c-971b-0fa3b75fd468-openshift-service-ca\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.153066 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.153012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-metrics-certs\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.153066 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:01.153031    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-serving-cert: secret "maas-api-e2e-ait-abd9a747-serving-cert" not found
Sep 08 18:37:01.153280 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:01.153088    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls podName:62ebe370-9531-4a0c-971b-0fa3b75fd468 nodeName:}" failed. No retries permitted until 2026-09-08 18:37:01.653072229 +0000 UTC m=+804.063382292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls") pod "maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2" (UID: "62ebe370-9531-4a0c-971b-0fa3b75fd468") : secret "maas-api-e2e-ait-abd9a747-serving-cert" not found
Sep 08 18:37:01.153802 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.153779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ebe370-9531-4a0c-971b-0fa3b75fd468-openshift-service-ca\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.155712 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.155694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-metrics-certs\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.162072 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.162053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lct4d\" (UniqueName: \"kubernetes.io/projected/62ebe370-9531-4a0c-971b-0fa3b75fd468-kube-api-access-lct4d\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.658553 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.658512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.660988 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.660958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls\") pod \"maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:01.695036 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.695004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:37:01.875789 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:01.875494    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:02.160240 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.160211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"]
Sep 08 18:37:02.162326 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:37:02.162296    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ebe370_9531_4a0c_971b_0fa3b75fd468.slice/crio-1ebacee66c2fc163c31b81bddd434e0af84cc0480732c70f475f529f65cfe240 WatchSource:0}: Error finding container 1ebacee66c2fc163c31b81bddd434e0af84cc0480732c70f475f529f65cfe240: Status 404 returned error can't find the container with id 1ebacee66c2fc163c31b81bddd434e0af84cc0480732c70f475f529f65cfe240
Sep 08 18:37:02.723076 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.723027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2" event={"ID":"62ebe370-9531-4a0c-971b-0fa3b75fd468","Type":"ContainerStarted","Data":"ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59"}
Sep 08 18:37:02.723249 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.723089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:02.723249 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.723114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2" event={"ID":"62ebe370-9531-4a0c-971b-0fa3b75fd468","Type":"ContainerStarted","Data":"1ebacee66c2fc163c31b81bddd434e0af84cc0480732c70f475f529f65cfe240"}
Sep 08 18:37:02.724167 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.724145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"]
Sep 08 18:37:02.727947 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.727921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.730457 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.730420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-0eaf99-serving-cert\""
Sep 08 18:37:02.740458 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.740371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"]
Sep 08 18:37:02.743556 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.742623    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2" podStartSLOduration=2.289177515 podStartE2EDuration="2.742603787s" totalImagesPullingTime="453.426272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:37:02.740560865 +0000 UTC m=+805.150870949" watchObservedRunningTime="2026-09-08 18:37:02.742603787 +0000 UTC m=+805.152913870"
Sep 08 18:37:02.869146 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.869056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/241727fe-dea8-4e4d-9980-336f4cf9e352-openshift-service-ca\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.869146 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.869110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsn9f\" (UniqueName: \"kubernetes.io/projected/241727fe-dea8-4e4d-9980-336f4cf9e352-kube-api-access-tsn9f\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.869353 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.869213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-metrics-certs\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.869353 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.869260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.970280 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.970246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsn9f\" (UniqueName: \"kubernetes.io/projected/241727fe-dea8-4e4d-9980-336f4cf9e352-kube-api-access-tsn9f\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.970280 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.970289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-metrics-certs\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.970558 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.970311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.970558 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.970421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/241727fe-dea8-4e4d-9980-336f4cf9e352-openshift-service-ca\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.970558 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:02.970507    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-serving-cert: secret "maas-api-e2e-autores-0eaf99-serving-cert" not found
Sep 08 18:37:02.970723 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:02.970579    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls podName:241727fe-dea8-4e4d-9980-336f4cf9e352 nodeName:}" failed. No retries permitted until 2026-09-08 18:37:03.470558373 +0000 UTC m=+805.880868434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls") pod "maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk" (UID: "241727fe-dea8-4e4d-9980-336f4cf9e352") : secret "maas-api-e2e-autores-0eaf99-serving-cert" not found
Sep 08 18:37:02.971139 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.971120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/241727fe-dea8-4e4d-9980-336f4cf9e352-openshift-service-ca\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.973375 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.973347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-metrics-certs\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:02.981178 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:02.981157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsn9f\" (UniqueName: \"kubernetes.io/projected/241727fe-dea8-4e4d-9980-336f4cf9e352-kube-api-access-tsn9f\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:03.476598 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:03.476566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:03.479388 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:03.479349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls\") pod \"maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:03.640717 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:03.640681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:03.919579 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:03.919546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"]
Sep 08 18:37:03.923047 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:37:03.923017    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241727fe_dea8_4e4d_9980_336f4cf9e352.slice/crio-4b970f6b2329076f64bb5f1cff70045272b9c62b214d8b2b1752e9ea292e57f7 WatchSource:0}: Error finding container 4b970f6b2329076f64bb5f1cff70045272b9c62b214d8b2b1752e9ea292e57f7: Status 404 returned error can't find the container with id 4b970f6b2329076f64bb5f1cff70045272b9c62b214d8b2b1752e9ea292e57f7
Sep 08 18:37:04.735605 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:04.735569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk" event={"ID":"241727fe-dea8-4e4d-9980-336f4cf9e352","Type":"ContainerStarted","Data":"3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7"}
Sep 08 18:37:04.736095 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:04.735645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk" event={"ID":"241727fe-dea8-4e4d-9980-336f4cf9e352","Type":"ContainerStarted","Data":"4b970f6b2329076f64bb5f1cff70045272b9c62b214d8b2b1752e9ea292e57f7"}
Sep 08 18:37:04.736095 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:04.735680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:04.781360 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:04.781305    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk" podStartSLOduration=2.426452215 podStartE2EDuration="2.781288155s" totalImagesPullingTime="354.83594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:37:04.767201434 +0000 UTC m=+807.177511516" watchObservedRunningTime="2026-09-08 18:37:04.781288155 +0000 UTC m=+807.191598237"
Sep 08 18:37:08.733884 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:08.733850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:10.746876 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:10.746845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:37:24.827707 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.827667    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"]
Sep 08 18:37:24.831215 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.831192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.833818 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.833797    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-15d796-serving-cert\""
Sep 08 18:37:24.840999 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.840962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"]
Sep 08 18:37:24.877359 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.877324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-metrics-certs\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.877563 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.877378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.877563 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.877405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gndxn\" (UniqueName: \"kubernetes.io/projected/0283aa88-872c-4e46-8a93-20c98cdb4a2a-kube-api-access-gndxn\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.877563 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.877529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283aa88-872c-4e46-8a93-20c98cdb4a2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.978939 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.978904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-metrics-certs\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.978939 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.978948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.979210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.978970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gndxn\" (UniqueName: \"kubernetes.io/projected/0283aa88-872c-4e46-8a93-20c98cdb4a2a-kube-api-access-gndxn\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.979210 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.979037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283aa88-872c-4e46-8a93-20c98cdb4a2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.979210 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:24.979077    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-serving-cert: secret "maas-api-e2e-inf-b-15d796-serving-cert" not found
Sep 08 18:37:24.979210 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:24.979155    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls podName:0283aa88-872c-4e46-8a93-20c98cdb4a2a nodeName:}" failed. No retries permitted until 2026-09-08 18:37:25.479136302 +0000 UTC m=+827.889446364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls") pod "maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5" (UID: "0283aa88-872c-4e46-8a93-20c98cdb4a2a") : secret "maas-api-e2e-inf-b-15d796-serving-cert" not found
Sep 08 18:37:24.979755 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.979737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283aa88-872c-4e46-8a93-20c98cdb4a2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.981532 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.981510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-metrics-certs\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:24.990186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:24.990166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gndxn\" (UniqueName: \"kubernetes.io/projected/0283aa88-872c-4e46-8a93-20c98cdb4a2a-kube-api-access-gndxn\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:25.484809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:25.484769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:25.487559 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:25.487532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:25.742855 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:25.742768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:26.020263 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:26.020243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"]
Sep 08 18:37:26.022638 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:37:26.022612    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0283aa88_872c_4e46_8a93_20c98cdb4a2a.slice/crio-3431b7718c4a829f9a6ecd120dd8aec40d449b099c64cfd002fdac38665cfccd WatchSource:0}: Error finding container 3431b7718c4a829f9a6ecd120dd8aec40d449b099c64cfd002fdac38665cfccd: Status 404 returned error can't find the container with id 3431b7718c4a829f9a6ecd120dd8aec40d449b099c64cfd002fdac38665cfccd
Sep 08 18:37:26.854464 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:26.854373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5" event={"ID":"0283aa88-872c-4e46-8a93-20c98cdb4a2a","Type":"ContainerStarted","Data":"6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038"}
Sep 08 18:37:26.854464 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:26.854420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5" event={"ID":"0283aa88-872c-4e46-8a93-20c98cdb4a2a","Type":"ContainerStarted","Data":"3431b7718c4a829f9a6ecd120dd8aec40d449b099c64cfd002fdac38665cfccd"}
Sep 08 18:37:26.854635 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:26.854550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:26.874994 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:26.874941    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5" podStartSLOduration=2.441128551 podStartE2EDuration="2.874927053s" totalImagesPullingTime="433.798502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:37:26.872759627 +0000 UTC m=+829.283069699" watchObservedRunningTime="2026-09-08 18:37:26.874927053 +0000 UTC m=+829.285237130"
Sep 08 18:37:32.865772 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:32.865736    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:37:41.983835 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:41.983786    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"]
Sep 08 18:37:41.984294 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:41.984088    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2" podUID="62ebe370-9531-4a0c-971b-0fa3b75fd468" containerName="maas-api" containerID="cri-o://ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59" gracePeriod=30
Sep 08 18:37:42.227858 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.227832    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:42.346800 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.346705    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lct4d\" (UniqueName: \"kubernetes.io/projected/62ebe370-9531-4a0c-971b-0fa3b75fd468-kube-api-access-lct4d\") pod \"62ebe370-9531-4a0c-971b-0fa3b75fd468\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") "
Sep 08 18:37:42.346961 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.346805    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-metrics-certs\") pod \"62ebe370-9531-4a0c-971b-0fa3b75fd468\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") "
Sep 08 18:37:42.346961 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.346837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls\") pod \"62ebe370-9531-4a0c-971b-0fa3b75fd468\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") "
Sep 08 18:37:42.347087 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.346967    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ebe370-9531-4a0c-971b-0fa3b75fd468-openshift-service-ca\") pod \"62ebe370-9531-4a0c-971b-0fa3b75fd468\" (UID: \"62ebe370-9531-4a0c-971b-0fa3b75fd468\") "
Sep 08 18:37:42.347631 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.347588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ebe370-9531-4a0c-971b-0fa3b75fd468-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62ebe370-9531-4a0c-971b-0fa3b75fd468" (UID: "62ebe370-9531-4a0c-971b-0fa3b75fd468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:37:42.350540 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.350509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62ebe370-9531-4a0c-971b-0fa3b75fd468" (UID: "62ebe370-9531-4a0c-971b-0fa3b75fd468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:37:42.350686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.350538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62ebe370-9531-4a0c-971b-0fa3b75fd468" (UID: "62ebe370-9531-4a0c-971b-0fa3b75fd468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:37:42.350686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.350576    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ebe370-9531-4a0c-971b-0fa3b75fd468-kube-api-access-lct4d" (OuterVolumeSpecName: "kube-api-access-lct4d") pod "62ebe370-9531-4a0c-971b-0fa3b75fd468" (UID: "62ebe370-9531-4a0c-971b-0fa3b75fd468"). InnerVolumeSpecName "kube-api-access-lct4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:37:42.448364 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.448331    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lct4d\" (UniqueName: \"kubernetes.io/projected/62ebe370-9531-4a0c-971b-0fa3b75fd468-kube-api-access-lct4d\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:37:42.448364 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.448359    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:37:42.448364 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.448368    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ebe370-9531-4a0c-971b-0fa3b75fd468-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:37:42.448617 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.448376    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ebe370-9531-4a0c-971b-0fa3b75fd468-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:37:42.944960 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.944925    2580 generic.go:356] "Generic (PLEG): container finished" podID="62ebe370-9531-4a0c-971b-0fa3b75fd468" containerID="ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59" exitCode=0
Sep 08 18:37:42.945124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.944968    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"
Sep 08 18:37:42.945124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.945011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2" event={"ID":"62ebe370-9531-4a0c-971b-0fa3b75fd468","Type":"ContainerDied","Data":"ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59"}
Sep 08 18:37:42.945124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.945061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2" event={"ID":"62ebe370-9531-4a0c-971b-0fa3b75fd468","Type":"ContainerDied","Data":"1ebacee66c2fc163c31b81bddd434e0af84cc0480732c70f475f529f65cfe240"}
Sep 08 18:37:42.945124 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.945083    2580 scope.go:117] "RemoveContainer" containerID="ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59"
Sep 08 18:37:42.954299 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.954282    2580 scope.go:117] "RemoveContainer" containerID="ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59"
Sep 08 18:37:42.954611 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:42.954593    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59\": container with ID starting with ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59 not found: ID does not exist" containerID="ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59"
Sep 08 18:37:42.954686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.954620    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59"} err="failed to get container status \"ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59\": rpc error: code = NotFound desc = could not find container \"ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59\": container with ID starting with ea7db5b050bf6077556c405acc76f3d25e2a8621724c54178c0752c6d47cca59 not found: ID does not exist"
Sep 08 18:37:42.966151 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.966131    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"]
Sep 08 18:37:42.970275 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:42.970258    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abd9a747-6bf8bb9f77-zk6j2"]
Sep 08 18:37:44.120858 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:44.120825    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ebe370-9531-4a0c-971b-0fa3b75fd468" path="/var/lib/kubelet/pods/62ebe370-9531-4a0c-971b-0fa3b75fd468/volumes"
Sep 08 18:37:46.421838 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.421801    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"]
Sep 08 18:37:46.422316 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.422198    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ebe370-9531-4a0c-971b-0fa3b75fd468" containerName="maas-api"
Sep 08 18:37:46.422316 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.422212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ebe370-9531-4a0c-971b-0fa3b75fd468" containerName="maas-api"
Sep 08 18:37:46.422316 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.422288    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ebe370-9531-4a0c-971b-0fa3b75fd468" containerName="maas-api"
Sep 08 18:37:46.426699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.426679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.429465 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.429442    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ad9a714c-serving-cert\""
Sep 08 18:37:46.436944 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.436913    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"]
Sep 08 18:37:46.587735 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.587696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.587914 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.587747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-metrics-certs\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.587914 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.587772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.587993 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.587948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbtf\" (UniqueName: \"kubernetes.io/projected/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-kube-api-access-gkbtf\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.689470 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.689350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbtf\" (UniqueName: \"kubernetes.io/projected/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-kube-api-access-gkbtf\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.689648 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.689505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.689648 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.689562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-metrics-certs\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.689648 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.689589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.689832 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:46.689813    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-serving-cert: secret "maas-api-e2e-migrate-ad9a714c-serving-cert" not found
Sep 08 18:37:46.689941 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:37:46.689918    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls podName:c8adc11f-d2ec-4e87-a17f-f4c30083b41c nodeName:}" failed. No retries permitted until 2026-09-08 18:37:47.189893222 +0000 UTC m=+849.600203285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls") pod "maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt" (UID: "c8adc11f-d2ec-4e87-a17f-f4c30083b41c") : secret "maas-api-e2e-migrate-ad9a714c-serving-cert" not found
Sep 08 18:37:46.690309 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.690282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.692522 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.692501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-metrics-certs\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:46.698383 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:46.698362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbtf\" (UniqueName: \"kubernetes.io/projected/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-kube-api-access-gkbtf\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:47.195513 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:47.195474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:47.198015 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:47.197979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls\") pod \"maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:47.339898 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:47.339864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:47.624809 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:47.624710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"]
Sep 08 18:37:47.626875 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:37:47.626845    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8adc11f_d2ec_4e87_a17f_f4c30083b41c.slice/crio-bee6bd4792465d0af58bd46a1eed9ed214f3c61a190947071898fe456bb38130 WatchSource:0}: Error finding container bee6bd4792465d0af58bd46a1eed9ed214f3c61a190947071898fe456bb38130: Status 404 returned error can't find the container with id bee6bd4792465d0af58bd46a1eed9ed214f3c61a190947071898fe456bb38130
Sep 08 18:37:47.973232 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:47.973205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt" event={"ID":"c8adc11f-d2ec-4e87-a17f-f4c30083b41c","Type":"ContainerStarted","Data":"bee6bd4792465d0af58bd46a1eed9ed214f3c61a190947071898fe456bb38130"}
Sep 08 18:37:48.978995 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:48.978961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt" event={"ID":"c8adc11f-d2ec-4e87-a17f-f4c30083b41c","Type":"ContainerStarted","Data":"b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4"}
Sep 08 18:37:48.979374 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:48.979169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:37:49.032693 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:49.032641    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt" podStartSLOduration=2.707564295 podStartE2EDuration="3.032625485s" totalImagesPullingTime="325.06119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:37:49.030098334 +0000 UTC m=+851.440408427" watchObservedRunningTime="2026-09-08 18:37:49.032625485 +0000 UTC m=+851.442935566"
Sep 08 18:37:54.990903 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:37:54.990867    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:38:17.679157 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:17.679120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"]
Sep 08 18:38:17.679697 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:17.679375    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt" podUID="c8adc11f-d2ec-4e87-a17f-f4c30083b41c" containerName="maas-api" containerID="cri-o://b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4" gracePeriod=30
Sep 08 18:38:17.918369 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:17.918346    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:38:18.004604 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.004530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-metrics-certs\") pod \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") "
Sep 08 18:38:18.004604 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.004594    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkbtf\" (UniqueName: \"kubernetes.io/projected/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-kube-api-access-gkbtf\") pod \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") "
Sep 08 18:38:18.004779 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.004652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls\") pod \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") "
Sep 08 18:38:18.004779 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.004730    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-openshift-service-ca\") pod \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\" (UID: \"c8adc11f-d2ec-4e87-a17f-f4c30083b41c\") "
Sep 08 18:38:18.005183 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.005152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8adc11f-d2ec-4e87-a17f-f4c30083b41c" (UID: "c8adc11f-d2ec-4e87-a17f-f4c30083b41c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:38:18.007507 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.007479    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8adc11f-d2ec-4e87-a17f-f4c30083b41c" (UID: "c8adc11f-d2ec-4e87-a17f-f4c30083b41c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:38:18.007613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.007562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-kube-api-access-gkbtf" (OuterVolumeSpecName: "kube-api-access-gkbtf") pod "c8adc11f-d2ec-4e87-a17f-f4c30083b41c" (UID: "c8adc11f-d2ec-4e87-a17f-f4c30083b41c"). InnerVolumeSpecName "kube-api-access-gkbtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:38:18.007613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.007574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c8adc11f-d2ec-4e87-a17f-f4c30083b41c" (UID: "c8adc11f-d2ec-4e87-a17f-f4c30083b41c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:38:18.105790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.105756    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkbtf\" (UniqueName: \"kubernetes.io/projected/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-kube-api-access-gkbtf\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:38:18.105790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.105783    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:38:18.105790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.105792    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:38:18.105790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.105799    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8adc11f-d2ec-4e87-a17f-f4c30083b41c-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:38:18.116407 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.116373    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8adc11f-d2ec-4e87-a17f-f4c30083b41c" containerID="b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4" exitCode=0
Sep 08 18:38:18.117863 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.117830    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"
Sep 08 18:38:18.121474 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.121440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt" event={"ID":"c8adc11f-d2ec-4e87-a17f-f4c30083b41c","Type":"ContainerDied","Data":"b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4"}
Sep 08 18:38:18.121595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.121494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt" event={"ID":"c8adc11f-d2ec-4e87-a17f-f4c30083b41c","Type":"ContainerDied","Data":"bee6bd4792465d0af58bd46a1eed9ed214f3c61a190947071898fe456bb38130"}
Sep 08 18:38:18.121595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.121525    2580 scope.go:117] "RemoveContainer" containerID="b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4"
Sep 08 18:38:18.131183 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.131169    2580 scope.go:117] "RemoveContainer" containerID="b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4"
Sep 08 18:38:18.131495 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:38:18.131477    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4\": container with ID starting with b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4 not found: ID does not exist" containerID="b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4"
Sep 08 18:38:18.131558 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.131504    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4"} err="failed to get container status \"b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4\": rpc error: code = NotFound desc = could not find container \"b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4\": container with ID starting with b748b81a3b9366bcfb6caaaf0b3bd4f609274a6c652deae2026301519be590b4 not found: ID does not exist"
Sep 08 18:38:18.146980 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.146958    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"]
Sep 08 18:38:18.151636 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:18.151608    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad9a714c-7cdc56dcf7-f45vt"]
Sep 08 18:38:20.120481 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:20.120449    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8adc11f-d2ec-4e87-a17f-f4c30083b41c" path="/var/lib/kubelet/pods/c8adc11f-d2ec-4e87-a17f-f4c30083b41c/volumes"
Sep 08 18:38:22.437394 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.437348    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"]
Sep 08 18:38:22.437985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.437828    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8adc11f-d2ec-4e87-a17f-f4c30083b41c" containerName="maas-api"
Sep 08 18:38:22.437985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.437842    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8adc11f-d2ec-4e87-a17f-f4c30083b41c" containerName="maas-api"
Sep 08 18:38:22.437985 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.437916    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8adc11f-d2ec-4e87-a17f-f4c30083b41c" containerName="maas-api"
Sep 08 18:38:22.443859 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.443828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.449277 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.449251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-12c5e22e-serving-cert\""
Sep 08 18:38:22.456109 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.456076    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"]
Sep 08 18:38:22.549764 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.549738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxccv\" (UniqueName: \"kubernetes.io/projected/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-kube-api-access-gxccv\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.549886 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.549778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-maas-api-tls\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.549983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.549900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-metrics-certs\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.549983 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.549963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-openshift-service-ca\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.651948 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.651905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxccv\" (UniqueName: \"kubernetes.io/projected/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-kube-api-access-gxccv\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.652292 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.651968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-maas-api-tls\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.652582 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.652548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-metrics-certs\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.652722 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.652612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-openshift-service-ca\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.653503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.653480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-openshift-service-ca\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.655625 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.655600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-maas-api-tls\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.655945 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.655925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-metrics-certs\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.661448 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.661412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxccv\" (UniqueName: \"kubernetes.io/projected/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-kube-api-access-gxccv\") pod \"maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:22.785947 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:22.785851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:23.066678 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:23.066652    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"]
Sep 08 18:38:23.069182 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:38:23.069157    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cf1900_ca8a_4750_aaaf_2f240a0bb37e.slice/crio-1e6dca4ce1a387ad01a11d0eb936b77b5dd933d6297ae243d6a88711cca9da1c WatchSource:0}: Error finding container 1e6dca4ce1a387ad01a11d0eb936b77b5dd933d6297ae243d6a88711cca9da1c: Status 404 returned error can't find the container with id 1e6dca4ce1a387ad01a11d0eb936b77b5dd933d6297ae243d6a88711cca9da1c
Sep 08 18:38:23.141465 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:23.141415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w" event={"ID":"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e","Type":"ContainerStarted","Data":"1e6dca4ce1a387ad01a11d0eb936b77b5dd933d6297ae243d6a88711cca9da1c"}
Sep 08 18:38:24.147713 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:24.147678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w" event={"ID":"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e","Type":"ContainerStarted","Data":"4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7"}
Sep 08 18:38:24.148103 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:24.147764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:24.167295 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:24.167244    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w" podStartSLOduration=1.750509587 podStartE2EDuration="2.167229588s" totalImagesPullingTime="416.720001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:24.16622669 +0000 UTC m=+886.576536775" watchObservedRunningTime="2026-09-08 18:38:24.167229588 +0000 UTC m=+886.577539709"
Sep 08 18:38:30.157443 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:30.157363    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:52.917799 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:52.917751    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"]
Sep 08 18:38:52.918284 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:52.918075    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w" podUID="c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" containerName="maas-api" containerID="cri-o://4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7" gracePeriod=30
Sep 08 18:38:53.180063 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.180006    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:53.238252 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.238226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-maas-api-tls\") pod \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") "
Sep 08 18:38:53.238252 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.238259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-openshift-service-ca\") pod \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") "
Sep 08 18:38:53.238574 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.238299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxccv\" (UniqueName: \"kubernetes.io/projected/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-kube-api-access-gxccv\") pod \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") "
Sep 08 18:38:53.238574 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.238344    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-metrics-certs\") pod \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\" (UID: \"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e\") "
Sep 08 18:38:53.238693 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.238668    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" (UID: "c1cf1900-ca8a-4750-aaaf-2f240a0bb37e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:38:53.240420 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.240396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" (UID: "c1cf1900-ca8a-4750-aaaf-2f240a0bb37e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:38:53.241108 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.241083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" (UID: "c1cf1900-ca8a-4750-aaaf-2f240a0bb37e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:38:53.241230 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.241102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-kube-api-access-gxccv" (OuterVolumeSpecName: "kube-api-access-gxccv") pod "c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" (UID: "c1cf1900-ca8a-4750-aaaf-2f240a0bb37e"). InnerVolumeSpecName "kube-api-access-gxccv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:38:53.292144 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.292116    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" containerID="4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7" exitCode=0
Sep 08 18:38:53.292351 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.292220    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"
Sep 08 18:38:53.292351 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.292230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w" event={"ID":"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e","Type":"ContainerDied","Data":"4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7"}
Sep 08 18:38:53.292351 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.292271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w" event={"ID":"c1cf1900-ca8a-4750-aaaf-2f240a0bb37e","Type":"ContainerDied","Data":"1e6dca4ce1a387ad01a11d0eb936b77b5dd933d6297ae243d6a88711cca9da1c"}
Sep 08 18:38:53.292351 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.292291    2580 scope.go:117] "RemoveContainer" containerID="4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7"
Sep 08 18:38:53.302503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.302482    2580 scope.go:117] "RemoveContainer" containerID="4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7"
Sep 08 18:38:53.302790 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:38:53.302771    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7\": container with ID starting with 4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7 not found: ID does not exist" containerID="4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7"
Sep 08 18:38:53.302867 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.302800    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7"} err="failed to get container status \"4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7\": rpc error: code = NotFound desc = could not find container \"4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7\": container with ID starting with 4e032729e2006f89230fc786e38939db84164dbff3edddd40541524822e1b8d7 not found: ID does not exist"
Sep 08 18:38:53.317595 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.317571    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"]
Sep 08 18:38:53.321116 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.321091    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12c5e22e-5b5ffb4477-42k6w"]
Sep 08 18:38:53.339892 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.339868    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxccv\" (UniqueName: \"kubernetes.io/projected/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-kube-api-access-gxccv\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:38:53.339892 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.339892    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:38:53.340017 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.339901    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:38:53.340017 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:53.339908    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:38:54.120962 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:54.120930    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" path="/var/lib/kubelet/pods/c1cf1900-ca8a-4750-aaaf-2f240a0bb37e/volumes"
Sep 08 18:38:58.811068 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.811026    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"]
Sep 08 18:38:58.811586 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.811565    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" containerName="maas-api"
Sep 08 18:38:58.811678 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.811589    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" containerName="maas-api"
Sep 08 18:38:58.811745 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.811719    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1cf1900-ca8a-4750-aaaf-2f240a0bb37e" containerName="maas-api"
Sep 08 18:38:58.816558 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.816532    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.819892 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.819864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-18913a0c-serving-cert\""
Sep 08 18:38:58.830146 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.830114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"]
Sep 08 18:38:58.894713 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.894680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b55c2-d80e-4cab-9005-c165544738c3-openshift-service-ca\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.894713 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.894722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-metrics-certs\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.894956 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.894832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6n25\" (UniqueName: \"kubernetes.io/projected/129b55c2-d80e-4cab-9005-c165544738c3-kube-api-access-l6n25\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.894956 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.894863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-maas-api-tls\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.995726 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.995690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b55c2-d80e-4cab-9005-c165544738c3-openshift-service-ca\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.995954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.995737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-metrics-certs\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.995954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.995841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6n25\" (UniqueName: \"kubernetes.io/projected/129b55c2-d80e-4cab-9005-c165544738c3-kube-api-access-l6n25\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.995954 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.995875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-maas-api-tls\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.996668 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.996640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b55c2-d80e-4cab-9005-c165544738c3-openshift-service-ca\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.999156 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.999131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-maas-api-tls\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:58.999518 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:58.999468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-metrics-certs\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:59.007325 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:59.007270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6n25\" (UniqueName: \"kubernetes.io/projected/129b55c2-d80e-4cab-9005-c165544738c3-kube-api-access-l6n25\") pod \"maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:59.128861 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:59.128822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:38:59.407013 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:38:59.406986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"]
Sep 08 18:38:59.409800 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:38:59.409768    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129b55c2_d80e_4cab_9005_c165544738c3.slice/crio-aa6c79118e2feb6aeac2c146c97b0cc5c2325006a83b16d1607de705b13d9920 WatchSource:0}: Error finding container aa6c79118e2feb6aeac2c146c97b0cc5c2325006a83b16d1607de705b13d9920: Status 404 returned error can't find the container with id aa6c79118e2feb6aeac2c146c97b0cc5c2325006a83b16d1607de705b13d9920
Sep 08 18:39:00.327357 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:00.327258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf" event={"ID":"129b55c2-d80e-4cab-9005-c165544738c3","Type":"ContainerStarted","Data":"72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65"}
Sep 08 18:39:00.327357 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:00.327317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf" event={"ID":"129b55c2-d80e-4cab-9005-c165544738c3","Type":"ContainerStarted","Data":"aa6c79118e2feb6aeac2c146c97b0cc5c2325006a83b16d1607de705b13d9920"}
Sep 08 18:39:00.327822 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:00.327464    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:39:00.345773 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:00.345716    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf" podStartSLOduration=1.874744419 podStartE2EDuration="2.345697829s" totalImagesPullingTime="470.95341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:00.344507724 +0000 UTC m=+922.754817806" watchObservedRunningTime="2026-09-08 18:39:00.345697829 +0000 UTC m=+922.756007910"
Sep 08 18:39:06.338097 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:06.338073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:39:11.920511 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:11.920475    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"]
Sep 08 18:39:11.924253 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:11.924234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:11.928374 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:11.928347    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1d8cb1\"/\"default-dockercfg-mczlp\""
Sep 08 18:39:11.928531 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:11.928387    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1d8cb1\"/\"openshift-service-ca.crt\""
Sep 08 18:39:11.928531 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:11.928386    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1d8cb1\"/\"test-model-1d8cb1-kserve-self-signed-certs\""
Sep 08 18:39:11.928531 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:11.928414    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1d8cb1\"/\"kube-root-ca.crt\""
Sep 08 18:39:11.933608 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:11.933589    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"]
Sep 08 18:39:12.023373 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.023339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-dshm\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.023592 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.023382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45crb\" (UniqueName: \"kubernetes.io/projected/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-kube-api-access-45crb\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.023592 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.023419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-model-cache\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.023592 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.023496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-home\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.023592 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.023518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tls-certs\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.023592 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.023550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tmp-dir\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.124863 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.124831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45crb\" (UniqueName: \"kubernetes.io/projected/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-kube-api-access-45crb\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.124863 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.124870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-model-cache\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.125151 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.125042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-home\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.125151 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.125071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tls-certs\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.125151 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.125100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tmp-dir\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.125304 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.125192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-dshm\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.125304 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.125252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-model-cache\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.125384 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.125351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-home\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.125584 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.125566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tmp-dir\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.127477 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.127458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-dshm\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.127672 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.127655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tls-certs\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.133745 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.133723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45crb\" (UniqueName: \"kubernetes.io/projected/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-kube-api-access-45crb\") pod \"test-model-1d8cb1-kserve-7f5f964d59-2687x\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") " pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.246695 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.246613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:12.519469 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:12.519376    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"]
Sep 08 18:39:12.521942 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:39:12.521898    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d39de74_9d22_4351_a7c4_1c6fc51e6ddc.slice/crio-0eac45c28d1ce6fdbfdad1a4209649326d22672ae5cdcf44f008f5bb3e16ac3b WatchSource:0}: Error finding container 0eac45c28d1ce6fdbfdad1a4209649326d22672ae5cdcf44f008f5bb3e16ac3b: Status 404 returned error can't find the container with id 0eac45c28d1ce6fdbfdad1a4209649326d22672ae5cdcf44f008f5bb3e16ac3b
Sep 08 18:39:13.396138 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:13.396097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x" event={"ID":"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc","Type":"ContainerStarted","Data":"af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff"}
Sep 08 18:39:13.396589 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:13.396149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x" event={"ID":"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc","Type":"ContainerStarted","Data":"0eac45c28d1ce6fdbfdad1a4209649326d22672ae5cdcf44f008f5bb3e16ac3b"}
Sep 08 18:39:13.414273 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:13.414224    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x" podStartSLOduration=2.414206747 podStartE2EDuration="2.414206747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:13.412532437 +0000 UTC m=+935.822842518" watchObservedRunningTime="2026-09-08 18:39:13.414206747 +0000 UTC m=+935.824516827"
Sep 08 18:39:22.247061 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:22.247001    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:22.249481 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:22.247081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:22.259738 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:22.259715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:22.451747 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:22.451715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:39:25.524397 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.524359    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"]
Sep 08 18:39:25.525239 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.524646    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk" podUID="241727fe-dea8-4e4d-9980-336f4cf9e352" containerName="maas-api" containerID="cri-o://3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7" gracePeriod=30
Sep 08 18:39:25.785573 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.785503    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:39:25.859533 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.859502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsn9f\" (UniqueName: \"kubernetes.io/projected/241727fe-dea8-4e4d-9980-336f4cf9e352-kube-api-access-tsn9f\") pod \"241727fe-dea8-4e4d-9980-336f4cf9e352\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") "
Sep 08 18:39:25.859724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.859574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/241727fe-dea8-4e4d-9980-336f4cf9e352-openshift-service-ca\") pod \"241727fe-dea8-4e4d-9980-336f4cf9e352\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") "
Sep 08 18:39:25.859724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.859606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-metrics-certs\") pod \"241727fe-dea8-4e4d-9980-336f4cf9e352\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") "
Sep 08 18:39:25.859724 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.859639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls\") pod \"241727fe-dea8-4e4d-9980-336f4cf9e352\" (UID: \"241727fe-dea8-4e4d-9980-336f4cf9e352\") "
Sep 08 18:39:25.860541 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.860000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241727fe-dea8-4e4d-9980-336f4cf9e352-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "241727fe-dea8-4e4d-9980-336f4cf9e352" (UID: "241727fe-dea8-4e4d-9980-336f4cf9e352"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:25.869169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.869102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241727fe-dea8-4e4d-9980-336f4cf9e352-kube-api-access-tsn9f" (OuterVolumeSpecName: "kube-api-access-tsn9f") pod "241727fe-dea8-4e4d-9980-336f4cf9e352" (UID: "241727fe-dea8-4e4d-9980-336f4cf9e352"). InnerVolumeSpecName "kube-api-access-tsn9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:25.869512 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.869481    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "241727fe-dea8-4e4d-9980-336f4cf9e352" (UID: "241727fe-dea8-4e4d-9980-336f4cf9e352"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:25.869722 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.869697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "241727fe-dea8-4e4d-9980-336f4cf9e352" (UID: "241727fe-dea8-4e4d-9980-336f4cf9e352"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:25.961172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.961135    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsn9f\" (UniqueName: \"kubernetes.io/projected/241727fe-dea8-4e4d-9980-336f4cf9e352-kube-api-access-tsn9f\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:39:25.961172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.961165    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/241727fe-dea8-4e4d-9980-336f4cf9e352-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:39:25.961172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.961177    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:39:25.961422 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:25.961185    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/241727fe-dea8-4e4d-9980-336f4cf9e352-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:39:26.461775 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.461689    2580 generic.go:356] "Generic (PLEG): container finished" podID="241727fe-dea8-4e4d-9980-336f4cf9e352" containerID="3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7" exitCode=0
Sep 08 18:39:26.461775 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.461753    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"
Sep 08 18:39:26.461968 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.461771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk" event={"ID":"241727fe-dea8-4e4d-9980-336f4cf9e352","Type":"ContainerDied","Data":"3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7"}
Sep 08 18:39:26.461968 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.461842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk" event={"ID":"241727fe-dea8-4e4d-9980-336f4cf9e352","Type":"ContainerDied","Data":"4b970f6b2329076f64bb5f1cff70045272b9c62b214d8b2b1752e9ea292e57f7"}
Sep 08 18:39:26.461968 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.461872    2580 scope.go:117] "RemoveContainer" containerID="3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7"
Sep 08 18:39:26.474387 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.474331    2580 scope.go:117] "RemoveContainer" containerID="3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7"
Sep 08 18:39:26.475044 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:39:26.475014    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7\": container with ID starting with 3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7 not found: ID does not exist" containerID="3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7"
Sep 08 18:39:26.475193 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.475051    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7"} err="failed to get container status \"3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7\": rpc error: code = NotFound desc = could not find container \"3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7\": container with ID starting with 3e508f3f295a24424d1ceb17437f4be3812d6082202d8e0b5304afe13a611ac7 not found: ID does not exist"
Sep 08 18:39:26.480636 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.480552    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"]
Sep 08 18:39:26.483171 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:26.483144    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk"]
Sep 08 18:39:28.120363 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:28.120329    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="241727fe-dea8-4e4d-9980-336f4cf9e352" path="/var/lib/kubelet/pods/241727fe-dea8-4e4d-9980-336f4cf9e352/volumes"
Sep 08 18:39:28.742450 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:28.742374    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0eaf99-7477b4597c-p9hmk" podUID="241727fe-dea8-4e4d-9980-336f4cf9e352" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.40:8443/health\": context deadline exceeded"
Sep 08 18:39:30.133311 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.133278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"]
Sep 08 18:39:30.133858 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.133534    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf" podUID="129b55c2-d80e-4cab-9005-c165544738c3" containerName="maas-api" containerID="cri-o://72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65" gracePeriod=30
Sep 08 18:39:30.400485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.400460    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:39:30.487172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.487134    2580 generic.go:356] "Generic (PLEG): container finished" podID="129b55c2-d80e-4cab-9005-c165544738c3" containerID="72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65" exitCode=0
Sep 08 18:39:30.487357 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.487225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf" event={"ID":"129b55c2-d80e-4cab-9005-c165544738c3","Type":"ContainerDied","Data":"72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65"}
Sep 08 18:39:30.487357 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.487276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf" event={"ID":"129b55c2-d80e-4cab-9005-c165544738c3","Type":"ContainerDied","Data":"aa6c79118e2feb6aeac2c146c97b0cc5c2325006a83b16d1607de705b13d9920"}
Sep 08 18:39:30.487357 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.487305    2580 scope.go:117] "RemoveContainer" containerID="72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65"
Sep 08 18:39:30.487555 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.487456    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"
Sep 08 18:39:30.497784 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.497762    2580 scope.go:117] "RemoveContainer" containerID="72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65"
Sep 08 18:39:30.498137 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:39:30.498108    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65\": container with ID starting with 72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65 not found: ID does not exist" containerID="72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65"
Sep 08 18:39:30.498212 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.498148    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65"} err="failed to get container status \"72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65\": rpc error: code = NotFound desc = could not find container \"72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65\": container with ID starting with 72302204a56efcdc767ca68b5c21f629a3c906b2e4c31e7d0ee26a3b87b8ae65 not found: ID does not exist"
Sep 08 18:39:30.509557 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.509532    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b55c2-d80e-4cab-9005-c165544738c3-openshift-service-ca\") pod \"129b55c2-d80e-4cab-9005-c165544738c3\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") "
Sep 08 18:39:30.509692 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.509586    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6n25\" (UniqueName: \"kubernetes.io/projected/129b55c2-d80e-4cab-9005-c165544738c3-kube-api-access-l6n25\") pod \"129b55c2-d80e-4cab-9005-c165544738c3\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") "
Sep 08 18:39:30.509692 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.509610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-metrics-certs\") pod \"129b55c2-d80e-4cab-9005-c165544738c3\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") "
Sep 08 18:39:30.509692 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.509650    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-maas-api-tls\") pod \"129b55c2-d80e-4cab-9005-c165544738c3\" (UID: \"129b55c2-d80e-4cab-9005-c165544738c3\") "
Sep 08 18:39:30.509982 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.509956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129b55c2-d80e-4cab-9005-c165544738c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "129b55c2-d80e-4cab-9005-c165544738c3" (UID: "129b55c2-d80e-4cab-9005-c165544738c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:30.512037 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.512012    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "129b55c2-d80e-4cab-9005-c165544738c3" (UID: "129b55c2-d80e-4cab-9005-c165544738c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:30.512188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.512167    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "129b55c2-d80e-4cab-9005-c165544738c3" (UID: "129b55c2-d80e-4cab-9005-c165544738c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:30.512242 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.512230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129b55c2-d80e-4cab-9005-c165544738c3-kube-api-access-l6n25" (OuterVolumeSpecName: "kube-api-access-l6n25") pod "129b55c2-d80e-4cab-9005-c165544738c3" (UID: "129b55c2-d80e-4cab-9005-c165544738c3"). InnerVolumeSpecName "kube-api-access-l6n25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:30.610750 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.610721    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.610750 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.610746    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129b55c2-d80e-4cab-9005-c165544738c3-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.610750 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.610755    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b55c2-d80e-4cab-9005-c165544738c3-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.610984 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.610764    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6n25\" (UniqueName: \"kubernetes.io/projected/129b55c2-d80e-4cab-9005-c165544738c3-kube-api-access-l6n25\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.809572 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.809535    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"]
Sep 08 18:39:30.814003 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:30.813975    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18913a0c-86d5f64cb7-spztf"]
Sep 08 18:39:32.120737 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:32.120698    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="129b55c2-d80e-4cab-9005-c165544738c3" path="/var/lib/kubelet/pods/129b55c2-d80e-4cab-9005-c165544738c3/volumes"
Sep 08 18:39:42.538401 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.538369    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"]
Sep 08 18:39:42.538796 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.538777    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129b55c2-d80e-4cab-9005-c165544738c3" containerName="maas-api"
Sep 08 18:39:42.538796 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.538791    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="129b55c2-d80e-4cab-9005-c165544738c3" containerName="maas-api"
Sep 08 18:39:42.538869 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.538816    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241727fe-dea8-4e4d-9980-336f4cf9e352" containerName="maas-api"
Sep 08 18:39:42.538869 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.538826    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="241727fe-dea8-4e4d-9980-336f4cf9e352" containerName="maas-api"
Sep 08 18:39:42.538937 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.538888    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="129b55c2-d80e-4cab-9005-c165544738c3" containerName="maas-api"
Sep 08 18:39:42.538937 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.538903    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="241727fe-dea8-4e4d-9980-336f4cf9e352" containerName="maas-api"
Sep 08 18:39:42.543309 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.543282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.546170 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.546144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c5ee49-serving-cert\""
Sep 08 18:39:42.554463 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.554418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"]
Sep 08 18:39:42.731890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.731850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-metrics-certs\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.732071 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.731898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-maas-api-tls\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.732071 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.731929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9341896-c015-44bf-a0da-5ff82ab2d454-openshift-service-ca\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.732071 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.731981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mbk\" (UniqueName: \"kubernetes.io/projected/b9341896-c015-44bf-a0da-5ff82ab2d454-kube-api-access-77mbk\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.832875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.832798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mbk\" (UniqueName: \"kubernetes.io/projected/b9341896-c015-44bf-a0da-5ff82ab2d454-kube-api-access-77mbk\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.833050 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.832888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-metrics-certs\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.833050 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.832916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-maas-api-tls\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.833050 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.832936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9341896-c015-44bf-a0da-5ff82ab2d454-openshift-service-ca\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.833610 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.833590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9341896-c015-44bf-a0da-5ff82ab2d454-openshift-service-ca\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.835442 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.835395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-maas-api-tls\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.835586 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.835564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-metrics-certs\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.840870 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.840846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mbk\" (UniqueName: \"kubernetes.io/projected/b9341896-c015-44bf-a0da-5ff82ab2d454-kube-api-access-77mbk\") pod \"maas-api-e2e-explicit-c5ee49-956746cd9-bsswr\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:42.856942 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:42.856908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:43.133022 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:43.132948    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"]
Sep 08 18:39:43.135222 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:39:43.135193    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9341896_c015_44bf_a0da_5ff82ab2d454.slice/crio-13bd50daaeaa328567939b57f2f561637198b6efa7a8a7cb20c9369236dfb5b5 WatchSource:0}: Error finding container 13bd50daaeaa328567939b57f2f561637198b6efa7a8a7cb20c9369236dfb5b5: Status 404 returned error can't find the container with id 13bd50daaeaa328567939b57f2f561637198b6efa7a8a7cb20c9369236dfb5b5
Sep 08 18:39:43.562851 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:43.562741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr" event={"ID":"b9341896-c015-44bf-a0da-5ff82ab2d454","Type":"ContainerStarted","Data":"13bd50daaeaa328567939b57f2f561637198b6efa7a8a7cb20c9369236dfb5b5"}
Sep 08 18:39:44.569155 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:44.569113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr" event={"ID":"b9341896-c015-44bf-a0da-5ff82ab2d454","Type":"ContainerStarted","Data":"a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3"}
Sep 08 18:39:44.569599 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:44.569224    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:39:44.585920 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:44.585867    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr" podStartSLOduration=2.116658071 podStartE2EDuration="2.585846852s" totalImagesPullingTime="469.188781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:44.585470086 +0000 UTC m=+966.995780177" watchObservedRunningTime="2026-09-08 18:39:44.585846852 +0000 UTC m=+966.996156932"
Sep 08 18:39:50.579295 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:39:50.579263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:40:55.376917 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.376885    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"]
Sep 08 18:40:55.377554 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.377089    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr" podUID="b9341896-c015-44bf-a0da-5ff82ab2d454" containerName="maas-api" containerID="cri-o://a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3" gracePeriod=30
Sep 08 18:40:55.630678 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.630617    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:40:55.648093 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.648069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9341896-c015-44bf-a0da-5ff82ab2d454-openshift-service-ca\") pod \"b9341896-c015-44bf-a0da-5ff82ab2d454\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") "
Sep 08 18:40:55.648184 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.648097    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77mbk\" (UniqueName: \"kubernetes.io/projected/b9341896-c015-44bf-a0da-5ff82ab2d454-kube-api-access-77mbk\") pod \"b9341896-c015-44bf-a0da-5ff82ab2d454\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") "
Sep 08 18:40:55.648184 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.648139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-metrics-certs\") pod \"b9341896-c015-44bf-a0da-5ff82ab2d454\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") "
Sep 08 18:40:55.648291 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.648196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-maas-api-tls\") pod \"b9341896-c015-44bf-a0da-5ff82ab2d454\" (UID: \"b9341896-c015-44bf-a0da-5ff82ab2d454\") "
Sep 08 18:40:55.648492 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.648467    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9341896-c015-44bf-a0da-5ff82ab2d454-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9341896-c015-44bf-a0da-5ff82ab2d454" (UID: "b9341896-c015-44bf-a0da-5ff82ab2d454"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:55.650639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.650595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9341896-c015-44bf-a0da-5ff82ab2d454" (UID: "b9341896-c015-44bf-a0da-5ff82ab2d454"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:55.650639 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.650616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9341896-c015-44bf-a0da-5ff82ab2d454-kube-api-access-77mbk" (OuterVolumeSpecName: "kube-api-access-77mbk") pod "b9341896-c015-44bf-a0da-5ff82ab2d454" (UID: "b9341896-c015-44bf-a0da-5ff82ab2d454"). InnerVolumeSpecName "kube-api-access-77mbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:40:55.651138 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.651115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b9341896-c015-44bf-a0da-5ff82ab2d454" (UID: "b9341896-c015-44bf-a0da-5ff82ab2d454"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:40:55.749172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.749144    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:40:55.749172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.749169    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9341896-c015-44bf-a0da-5ff82ab2d454-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:40:55.749172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.749179    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9341896-c015-44bf-a0da-5ff82ab2d454-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:40:55.749396 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.749189    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mbk\" (UniqueName: \"kubernetes.io/projected/b9341896-c015-44bf-a0da-5ff82ab2d454-kube-api-access-77mbk\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:40:55.926930 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.926852    2580 generic.go:356] "Generic (PLEG): container finished" podID="b9341896-c015-44bf-a0da-5ff82ab2d454" containerID="a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3" exitCode=0
Sep 08 18:40:55.926930 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.926897    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"
Sep 08 18:40:55.927126 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.926936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr" event={"ID":"b9341896-c015-44bf-a0da-5ff82ab2d454","Type":"ContainerDied","Data":"a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3"}
Sep 08 18:40:55.927126 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.926994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr" event={"ID":"b9341896-c015-44bf-a0da-5ff82ab2d454","Type":"ContainerDied","Data":"13bd50daaeaa328567939b57f2f561637198b6efa7a8a7cb20c9369236dfb5b5"}
Sep 08 18:40:55.927126 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.927019    2580 scope.go:117] "RemoveContainer" containerID="a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3"
Sep 08 18:40:55.936216 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.936200    2580 scope.go:117] "RemoveContainer" containerID="a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3"
Sep 08 18:40:55.936524 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:40:55.936498    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3\": container with ID starting with a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3 not found: ID does not exist" containerID="a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3"
Sep 08 18:40:55.936590 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.936536    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3"} err="failed to get container status \"a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3\": rpc error: code = NotFound desc = could not find container \"a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3\": container with ID starting with a2a1005283d1004d7c5c06716fb054761b6d162e6e50ce2af02c3de7042db8d3 not found: ID does not exist"
Sep 08 18:40:55.948787 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.948766    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"]
Sep 08 18:40:55.950411 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:55.950392    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr"]
Sep 08 18:40:56.120955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:56.120926    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9341896-c015-44bf-a0da-5ff82ab2d454" path="/var/lib/kubelet/pods/b9341896-c015-44bf-a0da-5ff82ab2d454/volumes"
Sep 08 18:40:58.575472 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:40:58.575410    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c5ee49-956746cd9-bsswr" podUID="b9341896-c015-44bf-a0da-5ff82ab2d454" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 08 18:41:11.124405 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.124374    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"]
Sep 08 18:41:11.124949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.124779    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9341896-c015-44bf-a0da-5ff82ab2d454" containerName="maas-api"
Sep 08 18:41:11.124949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.124791    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9341896-c015-44bf-a0da-5ff82ab2d454" containerName="maas-api"
Sep 08 18:41:11.124949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.124886    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9341896-c015-44bf-a0da-5ff82ab2d454" containerName="maas-api"
Sep 08 18:41:11.128213 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.128190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.131018 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.130999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2e25b35c-serving-cert\""
Sep 08 18:41:11.141062 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.141034    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"]
Sep 08 18:41:11.196616 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.196579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-openshift-service-ca\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.196616 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.196619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.196847 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.196643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-metrics-certs\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.196847 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.196682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbhr\" (UniqueName: \"kubernetes.io/projected/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-kube-api-access-fjbhr\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.297944 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.297908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-openshift-service-ca\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.297944 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.297950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.298179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.297977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-metrics-certs\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.298179 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:41:11.298076    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-serving-cert: secret "maas-api-e2e-mt-2e25b35c-serving-cert" not found
Sep 08 18:41:11.298179 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:41:11.298151    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls podName:18ade2e3-8ffd-4a3f-b0e3-03d10997554c nodeName:}" failed. No retries permitted until 2026-09-08 18:41:11.798133628 +0000 UTC m=+1054.208443687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls") pod "maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t" (UID: "18ade2e3-8ffd-4a3f-b0e3-03d10997554c") : secret "maas-api-e2e-mt-2e25b35c-serving-cert" not found
Sep 08 18:41:11.298179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.298172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbhr\" (UniqueName: \"kubernetes.io/projected/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-kube-api-access-fjbhr\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.298810 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.298788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-openshift-service-ca\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.300370 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.300352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-metrics-certs\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.306788 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.306768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbhr\" (UniqueName: \"kubernetes.io/projected/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-kube-api-access-fjbhr\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.803496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.803455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:11.806629 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:11.806596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls\") pod \"maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:12.042817 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:12.042791    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:12.318313 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:12.318288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"]
Sep 08 18:41:12.320999 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:41:12.320967    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ade2e3_8ffd_4a3f_b0e3_03d10997554c.slice/crio-d839d477eb04e9e9effa1b991327671565df348d9bbbc100068dee237f2b5796 WatchSource:0}: Error finding container d839d477eb04e9e9effa1b991327671565df348d9bbbc100068dee237f2b5796: Status 404 returned error can't find the container with id d839d477eb04e9e9effa1b991327671565df348d9bbbc100068dee237f2b5796
Sep 08 18:41:12.322541 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:12.322522    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:41:13.010490 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:13.010393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t" event={"ID":"18ade2e3-8ffd-4a3f-b0e3-03d10997554c","Type":"ContainerStarted","Data":"99775dc5335338ef2da0348e4d2e6b4e9bcec12c553d2637611681ca494afae3"}
Sep 08 18:41:13.010490 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:13.010479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t" event={"ID":"18ade2e3-8ffd-4a3f-b0e3-03d10997554c","Type":"ContainerStarted","Data":"d839d477eb04e9e9effa1b991327671565df348d9bbbc100068dee237f2b5796"}
Sep 08 18:41:13.010686 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:13.010502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:13.041872 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:13.041820    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t" podStartSLOduration=1.650918697 podStartE2EDuration="2.0418015s" totalImagesPullingTime="390.882803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:41:13.038920409 +0000 UTC m=+1055.449230490" watchObservedRunningTime="2026-09-08 18:41:13.0418015 +0000 UTC m=+1055.452111580"
Sep 08 18:41:16.990839 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:16.990724    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"]
Sep 08 18:41:19.022045 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:19.022017    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:29.330300 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.330264    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"]
Sep 08 18:41:29.330802 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.330535    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z" podUID="46b0816e-9492-4fcb-9f34-11fff090173c" containerName="maas-api" containerID="cri-o://380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2" gracePeriod=30
Sep 08 18:41:29.584087 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.584025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:41:29.684187 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.684157    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcd5b\" (UniqueName: \"kubernetes.io/projected/46b0816e-9492-4fcb-9f34-11fff090173c-kube-api-access-vcd5b\") pod \"46b0816e-9492-4fcb-9f34-11fff090173c\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") "
Sep 08 18:41:29.684352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.684248    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-metrics-certs\") pod \"46b0816e-9492-4fcb-9f34-11fff090173c\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") "
Sep 08 18:41:29.684352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.684274    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b0816e-9492-4fcb-9f34-11fff090173c-openshift-service-ca\") pod \"46b0816e-9492-4fcb-9f34-11fff090173c\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") "
Sep 08 18:41:29.684352 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.684293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls\") pod \"46b0816e-9492-4fcb-9f34-11fff090173c\" (UID: \"46b0816e-9492-4fcb-9f34-11fff090173c\") "
Sep 08 18:41:29.684702 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.684674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b0816e-9492-4fcb-9f34-11fff090173c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46b0816e-9492-4fcb-9f34-11fff090173c" (UID: "46b0816e-9492-4fcb-9f34-11fff090173c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:29.686562 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.686536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "46b0816e-9492-4fcb-9f34-11fff090173c" (UID: "46b0816e-9492-4fcb-9f34-11fff090173c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:29.686672 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.686581    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46b0816e-9492-4fcb-9f34-11fff090173c" (UID: "46b0816e-9492-4fcb-9f34-11fff090173c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:29.686672 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.686583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b0816e-9492-4fcb-9f34-11fff090173c-kube-api-access-vcd5b" (OuterVolumeSpecName: "kube-api-access-vcd5b") pod "46b0816e-9492-4fcb-9f34-11fff090173c" (UID: "46b0816e-9492-4fcb-9f34-11fff090173c"). InnerVolumeSpecName "kube-api-access-vcd5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:29.785848 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.785813    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:29.785848 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.785840    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b0816e-9492-4fcb-9f34-11fff090173c-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:29.785848 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.785849    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b0816e-9492-4fcb-9f34-11fff090173c-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:29.786074 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:29.785859    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcd5b\" (UniqueName: \"kubernetes.io/projected/46b0816e-9492-4fcb-9f34-11fff090173c-kube-api-access-vcd5b\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:30.096746 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.096712    2580 generic.go:356] "Generic (PLEG): container finished" podID="46b0816e-9492-4fcb-9f34-11fff090173c" containerID="380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2" exitCode=0
Sep 08 18:41:30.096746 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.096749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z" event={"ID":"46b0816e-9492-4fcb-9f34-11fff090173c","Type":"ContainerDied","Data":"380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2"}
Sep 08 18:41:30.096973 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.096759    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"
Sep 08 18:41:30.096973 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.096814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z" event={"ID":"46b0816e-9492-4fcb-9f34-11fff090173c","Type":"ContainerDied","Data":"0e14fcf8723a05f13f0cc7fe1328ab0798669a9ebb7c1184c6259a0d10b2fb5d"}
Sep 08 18:41:30.096973 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.096835    2580 scope.go:117] "RemoveContainer" containerID="380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2"
Sep 08 18:41:30.106156 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.106135    2580 scope.go:117] "RemoveContainer" containerID="380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2"
Sep 08 18:41:30.106403 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:41:30.106385    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2\": container with ID starting with 380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2 not found: ID does not exist" containerID="380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2"
Sep 08 18:41:30.106474 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.106412    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2"} err="failed to get container status \"380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2\": rpc error: code = NotFound desc = could not find container \"380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2\": container with ID starting with 380d7197f069a24f48302e7261a876c2a918c87c197758d275e385d039c12ea2 not found: ID does not exist"
Sep 08 18:41:30.129866 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.129838    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"]
Sep 08 18:41:30.140190 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:30.140162    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1d8cb1-6ffb755fcd-5wd8z"]
Sep 08 18:41:32.014868 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.014823    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x" podUID="1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" containerName="main" containerID="cri-o://af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff" gracePeriod=45
Sep 08 18:41:32.120354 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.120321    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b0816e-9492-4fcb-9f34-11fff090173c" path="/var/lib/kubelet/pods/46b0816e-9492-4fcb-9f34-11fff090173c/volumes"
Sep 08 18:41:32.276542 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.276467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:41:32.313388 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.311463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tmp-dir\") pod \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") "
Sep 08 18:41:32.313388 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.311523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-model-cache\") pod \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") "
Sep 08 18:41:32.313388 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.311558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tls-certs\") pod \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") "
Sep 08 18:41:32.313388 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.311603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-home\") pod \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") "
Sep 08 18:41:32.313388 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.311681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-dshm\") pod \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") "
Sep 08 18:41:32.313388 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.311719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45crb\" (UniqueName: \"kubernetes.io/projected/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-kube-api-access-45crb\") pod \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\" (UID: \"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc\") "
Sep 08 18:41:32.316866 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.316826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-kube-api-access-45crb" (OuterVolumeSpecName: "kube-api-access-45crb") pod "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" (UID: "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc"). InnerVolumeSpecName "kube-api-access-45crb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:32.317071 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.317049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-home" (OuterVolumeSpecName: "home") pod "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" (UID: "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:32.319359 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.319321    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-dshm" (OuterVolumeSpecName: "dshm") pod "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" (UID: "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:32.319560 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.319541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" (UID: "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:32.319726 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.319708    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" (UID: "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:32.322940 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.322911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" (UID: "1d39de74-9d22-4351-a7c4-1c6fc51e6ddc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:32.415188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.415149    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tmp-dir\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:32.415188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.415184    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-model-cache\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:32.415188 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.415197    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-tls-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:32.415479 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.415209    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-home\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:32.415479 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.415223    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-dshm\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:32.415479 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:32.415235    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45crb\" (UniqueName: \"kubernetes.io/projected/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc-kube-api-access-45crb\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:33.116674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.116638    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" containerID="af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff" exitCode=0
Sep 08 18:41:33.117179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.116682    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"
Sep 08 18:41:33.117179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.116713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x" event={"ID":"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc","Type":"ContainerDied","Data":"af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff"}
Sep 08 18:41:33.117179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.116748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x" event={"ID":"1d39de74-9d22-4351-a7c4-1c6fc51e6ddc","Type":"ContainerDied","Data":"0eac45c28d1ce6fdbfdad1a4209649326d22672ae5cdcf44f008f5bb3e16ac3b"}
Sep 08 18:41:33.117179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.116765    2580 scope.go:117] "RemoveContainer" containerID="af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff"
Sep 08 18:41:33.128246 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.128227    2580 scope.go:117] "RemoveContainer" containerID="af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff"
Sep 08 18:41:33.128630 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:41:33.128606    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff\": container with ID starting with af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff not found: ID does not exist" containerID="af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff"
Sep 08 18:41:33.128682 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.128644    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff"} err="failed to get container status \"af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff\": rpc error: code = NotFound desc = could not find container \"af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff\": container with ID starting with af113cd58c683f5a26c0b65f93d9d47275ed84c64a6c190fa957a8d350d815ff not found: ID does not exist"
Sep 08 18:41:33.140757 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.140732    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"]
Sep 08 18:41:33.147008 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:33.146984    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1d8cb1/test-model-1d8cb1-kserve-7f5f964d59-2687x"]
Sep 08 18:41:34.121419 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:34.121381    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" path="/var/lib/kubelet/pods/1d39de74-9d22-4351-a7c4-1c6fc51e6ddc/volumes"
Sep 08 18:41:47.759618 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:47.759584    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"]
Sep 08 18:41:47.760184 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:47.759805    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5" podUID="0283aa88-872c-4e46-8a93-20c98cdb4a2a" containerName="maas-api" containerID="cri-o://6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038" gracePeriod=30
Sep 08 18:41:47.860140 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:47.860100    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5" podUID="0283aa88-872c-4e46-8a93-20c98cdb4a2a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.41:8443/health\": dial tcp 10.133.0.41:8443: connect: connection refused"
Sep 08 18:41:48.016555 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.016490    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:41:48.173470 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.173410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls\") pod \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") "
Sep 08 18:41:48.173651 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.173482    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gndxn\" (UniqueName: \"kubernetes.io/projected/0283aa88-872c-4e46-8a93-20c98cdb4a2a-kube-api-access-gndxn\") pod \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") "
Sep 08 18:41:48.173651 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.173524    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283aa88-872c-4e46-8a93-20c98cdb4a2a-openshift-service-ca\") pod \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") "
Sep 08 18:41:48.173651 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.173569    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-metrics-certs\") pod \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\" (UID: \"0283aa88-872c-4e46-8a93-20c98cdb4a2a\") "
Sep 08 18:41:48.173962 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.173935    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0283aa88-872c-4e46-8a93-20c98cdb4a2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0283aa88-872c-4e46-8a93-20c98cdb4a2a" (UID: "0283aa88-872c-4e46-8a93-20c98cdb4a2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:48.175766 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.175740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0283aa88-872c-4e46-8a93-20c98cdb4a2a" (UID: "0283aa88-872c-4e46-8a93-20c98cdb4a2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:48.176191 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.176173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0283aa88-872c-4e46-8a93-20c98cdb4a2a-kube-api-access-gndxn" (OuterVolumeSpecName: "kube-api-access-gndxn") pod "0283aa88-872c-4e46-8a93-20c98cdb4a2a" (UID: "0283aa88-872c-4e46-8a93-20c98cdb4a2a"). InnerVolumeSpecName "kube-api-access-gndxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:48.176277 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.176187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0283aa88-872c-4e46-8a93-20c98cdb4a2a" (UID: "0283aa88-872c-4e46-8a93-20c98cdb4a2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:48.187650 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.187623    2580 generic.go:356] "Generic (PLEG): container finished" podID="0283aa88-872c-4e46-8a93-20c98cdb4a2a" containerID="6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038" exitCode=0
Sep 08 18:41:48.187740 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.187675    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"
Sep 08 18:41:48.187740 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.187705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5" event={"ID":"0283aa88-872c-4e46-8a93-20c98cdb4a2a","Type":"ContainerDied","Data":"6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038"}
Sep 08 18:41:48.187813 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.187758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5" event={"ID":"0283aa88-872c-4e46-8a93-20c98cdb4a2a","Type":"ContainerDied","Data":"3431b7718c4a829f9a6ecd120dd8aec40d449b099c64cfd002fdac38665cfccd"}
Sep 08 18:41:48.187813 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.187779    2580 scope.go:117] "RemoveContainer" containerID="6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038"
Sep 08 18:41:48.196685 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.196670    2580 scope.go:117] "RemoveContainer" containerID="6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038"
Sep 08 18:41:48.196961 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:41:48.196944    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038\": container with ID starting with 6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038 not found: ID does not exist" containerID="6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038"
Sep 08 18:41:48.197014 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.196970    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038"} err="failed to get container status \"6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038\": rpc error: code = NotFound desc = could not find container \"6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038\": container with ID starting with 6eea9652c09d8893b04717d6f85bec0da0c04f36dbc206dbaced97c83f4c7038 not found: ID does not exist"
Sep 08 18:41:48.208619 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.208599    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"]
Sep 08 18:41:48.212816 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.212795    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15d796-86678ff5dd-mc4q5"]
Sep 08 18:41:48.275066 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.275000    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:48.275066 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.275021    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gndxn\" (UniqueName: \"kubernetes.io/projected/0283aa88-872c-4e46-8a93-20c98cdb4a2a-kube-api-access-gndxn\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:48.275066 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.275031    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283aa88-872c-4e46-8a93-20c98cdb4a2a-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:48.275066 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:48.275045    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0283aa88-872c-4e46-8a93-20c98cdb4a2a-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:50.120926 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:50.120894    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0283aa88-872c-4e46-8a93-20c98cdb4a2a" path="/var/lib/kubelet/pods/0283aa88-872c-4e46-8a93-20c98cdb4a2a/volumes"
Sep 08 18:41:53.067181 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.067151    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"]
Sep 08 18:41:53.067673 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.067464    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t" podUID="18ade2e3-8ffd-4a3f-b0e3-03d10997554c" containerName="maas-api" containerID="cri-o://99775dc5335338ef2da0348e4d2e6b4e9bcec12c553d2637611681ca494afae3" gracePeriod=30
Sep 08 18:41:53.214894 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.214822    2580 generic.go:356] "Generic (PLEG): container finished" podID="18ade2e3-8ffd-4a3f-b0e3-03d10997554c" containerID="99775dc5335338ef2da0348e4d2e6b4e9bcec12c553d2637611681ca494afae3" exitCode=0
Sep 08 18:41:53.214894 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.214881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t" event={"ID":"18ade2e3-8ffd-4a3f-b0e3-03d10997554c","Type":"ContainerDied","Data":"99775dc5335338ef2da0348e4d2e6b4e9bcec12c553d2637611681ca494afae3"}
Sep 08 18:41:53.316757 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.316732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:53.421085 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.421054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-openshift-service-ca\") pod \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") "
Sep 08 18:41:53.421269 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.421152    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls\") pod \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") "
Sep 08 18:41:53.421269 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.421184    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjbhr\" (UniqueName: \"kubernetes.io/projected/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-kube-api-access-fjbhr\") pod \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") "
Sep 08 18:41:53.421269 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.421229    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-metrics-certs\") pod \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\" (UID: \"18ade2e3-8ffd-4a3f-b0e3-03d10997554c\") "
Sep 08 18:41:53.421455 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.421415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18ade2e3-8ffd-4a3f-b0e3-03d10997554c" (UID: "18ade2e3-8ffd-4a3f-b0e3-03d10997554c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:53.423441 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.423399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-kube-api-access-fjbhr" (OuterVolumeSpecName: "kube-api-access-fjbhr") pod "18ade2e3-8ffd-4a3f-b0e3-03d10997554c" (UID: "18ade2e3-8ffd-4a3f-b0e3-03d10997554c"). InnerVolumeSpecName "kube-api-access-fjbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:53.423597 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.423490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18ade2e3-8ffd-4a3f-b0e3-03d10997554c" (UID: "18ade2e3-8ffd-4a3f-b0e3-03d10997554c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:53.423597 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.423589    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18ade2e3-8ffd-4a3f-b0e3-03d10997554c" (UID: "18ade2e3-8ffd-4a3f-b0e3-03d10997554c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:53.522472 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.522405    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:53.522472 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.522466    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:53.522472 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.522477    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbhr\" (UniqueName: \"kubernetes.io/projected/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-kube-api-access-fjbhr\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:53.522700 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:53.522485    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ade2e3-8ffd-4a3f-b0e3-03d10997554c-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:41:54.221242 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:54.221213    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"
Sep 08 18:41:54.221242 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:54.221237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t" event={"ID":"18ade2e3-8ffd-4a3f-b0e3-03d10997554c","Type":"ContainerDied","Data":"d839d477eb04e9e9effa1b991327671565df348d9bbbc100068dee237f2b5796"}
Sep 08 18:41:54.221701 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:54.221294    2580 scope.go:117] "RemoveContainer" containerID="99775dc5335338ef2da0348e4d2e6b4e9bcec12c553d2637611681ca494afae3"
Sep 08 18:41:54.238151 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:54.238128    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"]
Sep 08 18:41:54.241848 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:54.241827    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e25b35c-565696df7d-rmg8t"]
Sep 08 18:41:56.122137 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:56.122104    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ade2e3-8ffd-4a3f-b0e3-03d10997554c" path="/var/lib/kubelet/pods/18ade2e3-8ffd-4a3f-b0e3-03d10997554c/volumes"
Sep 08 18:41:59.572170 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572127    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"]
Sep 08 18:41:59.572600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572525    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" containerName="main"
Sep 08 18:41:59.572600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572537    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" containerName="main"
Sep 08 18:41:59.572600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572555    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b0816e-9492-4fcb-9f34-11fff090173c" containerName="maas-api"
Sep 08 18:41:59.572600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572561    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b0816e-9492-4fcb-9f34-11fff090173c" containerName="maas-api"
Sep 08 18:41:59.572600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572571    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0283aa88-872c-4e46-8a93-20c98cdb4a2a" containerName="maas-api"
Sep 08 18:41:59.572600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572586    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0283aa88-872c-4e46-8a93-20c98cdb4a2a" containerName="maas-api"
Sep 08 18:41:59.572600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572594    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ade2e3-8ffd-4a3f-b0e3-03d10997554c" containerName="maas-api"
Sep 08 18:41:59.572600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.572599    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ade2e3-8ffd-4a3f-b0e3-03d10997554c" containerName="maas-api"
Sep 08 18:41:59.573309 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.573285    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b0816e-9492-4fcb-9f34-11fff090173c" containerName="maas-api"
Sep 08 18:41:59.573513 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.573500    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d39de74-9d22-4351-a7c4-1c6fc51e6ddc" containerName="main"
Sep 08 18:41:59.573628 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.573618    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0283aa88-872c-4e46-8a93-20c98cdb4a2a" containerName="maas-api"
Sep 08 18:41:59.573705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.573696    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ade2e3-8ffd-4a3f-b0e3-03d10997554c" containerName="maas-api"
Sep 08 18:41:59.579368 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.579349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.584675 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.584644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-792c35-serving-cert\""
Sep 08 18:41:59.601186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.601152    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"]
Sep 08 18:41:59.684960 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.684927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.685144 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.685046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.685144 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.685080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzzhh\" (UniqueName: \"kubernetes.io/projected/e827d117-24c2-4d47-b054-b202f764b498-kube-api-access-pzzhh\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.685144 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.685111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e827d117-24c2-4d47-b054-b202f764b498-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.786205 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.786168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.786205 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.786220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzhh\" (UniqueName: \"kubernetes.io/projected/e827d117-24c2-4d47-b054-b202f764b498-kube-api-access-pzzhh\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.786523 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.786254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e827d117-24c2-4d47-b054-b202f764b498-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.786523 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.786309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.786523 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:41:59.786506    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-serving-cert: secret "maas-api-e2e-shared-a-w3-792c35-serving-cert" not found
Sep 08 18:41:59.786701 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:41:59.786575    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls podName:e827d117-24c2-4d47-b054-b202f764b498 nodeName:}" failed. No retries permitted until 2026-09-08 18:42:00.286554441 +0000 UTC m=+1102.696864518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls") pod "maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m" (UID: "e827d117-24c2-4d47-b054-b202f764b498") : secret "maas-api-e2e-shared-a-w3-792c35-serving-cert" not found
Sep 08 18:41:59.787097 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.787021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e827d117-24c2-4d47-b054-b202f764b498-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.788972 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.788949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:41:59.805420 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:41:59.805388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzzhh\" (UniqueName: \"kubernetes.io/projected/e827d117-24c2-4d47-b054-b202f764b498-kube-api-access-pzzhh\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:42:00.290915 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:00.290879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:42:00.293669 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:00.293641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:42:00.490473 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:00.490412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:42:00.776822 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:00.776793    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"]
Sep 08 18:42:00.779516 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:42:00.779487    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode827d117_24c2_4d47_b054_b202f764b498.slice/crio-fe9e81a4ff714914b0a389c76a3750b926774ee59541a5df9c2610dd23dd5635 WatchSource:0}: Error finding container fe9e81a4ff714914b0a389c76a3750b926774ee59541a5df9c2610dd23dd5635: Status 404 returned error can't find the container with id fe9e81a4ff714914b0a389c76a3750b926774ee59541a5df9c2610dd23dd5635
Sep 08 18:42:01.258927 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:01.258892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m" event={"ID":"e827d117-24c2-4d47-b054-b202f764b498","Type":"ContainerStarted","Data":"1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302"}
Sep 08 18:42:01.259091 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:01.258946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m" event={"ID":"e827d117-24c2-4d47-b054-b202f764b498","Type":"ContainerStarted","Data":"fe9e81a4ff714914b0a389c76a3750b926774ee59541a5df9c2610dd23dd5635"}
Sep 08 18:42:01.259091 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:01.258976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:42:01.280494 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:01.280420    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m" podStartSLOduration=1.946575686 podStartE2EDuration="2.280400462s" totalImagesPullingTime="333.824776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:01.279476103 +0000 UTC m=+1103.689786185" watchObservedRunningTime="2026-09-08 18:42:01.280400462 +0000 UTC m=+1103.690710543"
Sep 08 18:42:07.270960 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:07.270932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:42:18.633079 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.633042    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"]
Sep 08 18:42:18.636286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.636265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.639232 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.639211    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-3b04d6-serving-cert\""
Sep 08 18:42:18.650694 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.650660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"]
Sep 08 18:42:18.759789 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.759753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.759949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.759836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01719465-a7df-44de-ba36-6e170d86a468-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.759949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.759870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.759949 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.759919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5npw\" (UniqueName: \"kubernetes.io/projected/01719465-a7df-44de-ba36-6e170d86a468-kube-api-access-t5npw\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.860831 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.860790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01719465-a7df-44de-ba36-6e170d86a468-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.861005 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.860870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.861005 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.860929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5npw\" (UniqueName: \"kubernetes.io/projected/01719465-a7df-44de-ba36-6e170d86a468-kube-api-access-t5npw\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.861114 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.861007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.861114 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:42:18.861049    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-serving-cert: secret "maas-api-e2e-shared-a-w2-3b04d6-serving-cert" not found
Sep 08 18:42:18.861246 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:42:18.861124    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls podName:01719465-a7df-44de-ba36-6e170d86a468 nodeName:}" failed. No retries permitted until 2026-09-08 18:42:19.361102196 +0000 UTC m=+1121.771412268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls") pod "maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr" (UID: "01719465-a7df-44de-ba36-6e170d86a468") : secret "maas-api-e2e-shared-a-w2-3b04d6-serving-cert" not found
Sep 08 18:42:18.861647 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.861619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01719465-a7df-44de-ba36-6e170d86a468-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.863720 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.863701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:18.872040 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:18.872015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5npw\" (UniqueName: \"kubernetes.io/projected/01719465-a7df-44de-ba36-6e170d86a468-kube-api-access-t5npw\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:19.366536 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:19.366502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:19.369464 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:19.369407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:19.555726 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:19.555694    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:19.842497 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:19.842475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"]
Sep 08 18:42:19.846353 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:42:19.846319    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01719465_a7df_44de_ba36_6e170d86a468.slice/crio-0aac0d977da128a602b80f0b76fdf7722ec3f66a7be4630557b07ea752731a96 WatchSource:0}: Error finding container 0aac0d977da128a602b80f0b76fdf7722ec3f66a7be4630557b07ea752731a96: Status 404 returned error can't find the container with id 0aac0d977da128a602b80f0b76fdf7722ec3f66a7be4630557b07ea752731a96
Sep 08 18:42:20.356606 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:20.356563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr" event={"ID":"01719465-a7df-44de-ba36-6e170d86a468","Type":"ContainerStarted","Data":"5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80"}
Sep 08 18:42:20.356755 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:20.356630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr" event={"ID":"01719465-a7df-44de-ba36-6e170d86a468","Type":"ContainerStarted","Data":"0aac0d977da128a602b80f0b76fdf7722ec3f66a7be4630557b07ea752731a96"}
Sep 08 18:42:20.356755 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:20.356704    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:20.375692 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:20.375645    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr" podStartSLOduration=1.950203534 podStartE2EDuration="2.37562789s" totalImagesPullingTime="425.424356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:20.374037249 +0000 UTC m=+1122.784347332" watchObservedRunningTime="2026-09-08 18:42:20.37562789 +0000 UTC m=+1122.785937971"
Sep 08 18:42:26.367600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:26.367572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:42:28.517237 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.517203    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"]
Sep 08 18:42:28.521692 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.521669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.524881 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.524852    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-6255f2-serving-cert\""
Sep 08 18:42:28.534692 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.534666    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"]
Sep 08 18:42:28.658689 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.658652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.658689 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.658692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslhz\" (UniqueName: \"kubernetes.io/projected/2ac8d07e-7e9a-4596-a151-748fda456e9f-kube-api-access-fslhz\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.658924 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.658715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac8d07e-7e9a-4596-a151-748fda456e9f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.658924 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.658813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.760019 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.759980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.760209 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.760034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslhz\" (UniqueName: \"kubernetes.io/projected/2ac8d07e-7e9a-4596-a151-748fda456e9f-kube-api-access-fslhz\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.760209 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.760060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac8d07e-7e9a-4596-a151-748fda456e9f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.760209 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.760100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.760378 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:42:28.760236    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-serving-cert: secret "maas-api-e2e-shared-b-w3-6255f2-serving-cert" not found
Sep 08 18:42:28.760378 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:42:28.760298    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls podName:2ac8d07e-7e9a-4596-a151-748fda456e9f nodeName:}" failed. No retries permitted until 2026-09-08 18:42:29.260280856 +0000 UTC m=+1131.670590916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls") pod "maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r" (UID: "2ac8d07e-7e9a-4596-a151-748fda456e9f") : secret "maas-api-e2e-shared-b-w3-6255f2-serving-cert" not found
Sep 08 18:42:28.760831 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.760805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac8d07e-7e9a-4596-a151-748fda456e9f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.762629 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.762609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:28.777214 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:28.777152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslhz\" (UniqueName: \"kubernetes.io/projected/2ac8d07e-7e9a-4596-a151-748fda456e9f-kube-api-access-fslhz\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:29.265284 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:29.265252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:29.267894 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:29.267865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:29.435376 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:29.435337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:29.709727 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:29.709690    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"]
Sep 08 18:42:29.712577 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:42:29.712553    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac8d07e_7e9a_4596_a151_748fda456e9f.slice/crio-9e3a9023d5f8f15db1ad5944287b03920d1c25d5be658a5c0655f8ca27e0ccbb WatchSource:0}: Error finding container 9e3a9023d5f8f15db1ad5944287b03920d1c25d5be658a5c0655f8ca27e0ccbb: Status 404 returned error can't find the container with id 9e3a9023d5f8f15db1ad5944287b03920d1c25d5be658a5c0655f8ca27e0ccbb
Sep 08 18:42:30.408527 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:30.408496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r" event={"ID":"2ac8d07e-7e9a-4596-a151-748fda456e9f","Type":"ContainerStarted","Data":"582f97530b27fb97a3c7fff90a1f5e4777ba86b8ee1914a3d213fc692bcbdf80"}
Sep 08 18:42:30.408694 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:30.408547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r" event={"ID":"2ac8d07e-7e9a-4596-a151-748fda456e9f","Type":"ContainerStarted","Data":"9e3a9023d5f8f15db1ad5944287b03920d1c25d5be658a5c0655f8ca27e0ccbb"}
Sep 08 18:42:30.408694 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:30.408569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:30.428486 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:30.428414    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r" podStartSLOduration=2.092365968 podStartE2EDuration="2.428397237s" totalImagesPullingTime="336.031269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:30.426487267 +0000 UTC m=+1132.836797447" watchObservedRunningTime="2026-09-08 18:42:30.428397237 +0000 UTC m=+1132.838707342"
Sep 08 18:42:36.418989 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:36.418962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:42:46.717636 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.717596    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"]
Sep 08 18:42:46.722634 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.722609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.725739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.725711    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-951c3e-serving-cert\""
Sep 08 18:42:46.735915 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.735890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"]
Sep 08 18:42:46.738307 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.738283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.738477 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.738355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadffdcc-faae-4890-b359-c351cc5e8728-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.738477 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.738416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xz2t\" (UniqueName: \"kubernetes.io/projected/eadffdcc-faae-4890-b359-c351cc5e8728-kube-api-access-4xz2t\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.738477 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.738470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.839972 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.839922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.840165 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.840051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadffdcc-faae-4890-b359-c351cc5e8728-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.840165 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:42:46.840080    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-serving-cert: secret "maas-api-e2e-shared-b-w2-951c3e-serving-cert" not found
Sep 08 18:42:46.840165 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.840145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xz2t\" (UniqueName: \"kubernetes.io/projected/eadffdcc-faae-4890-b359-c351cc5e8728-kube-api-access-4xz2t\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.840315 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:42:46.840168    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls podName:eadffdcc-faae-4890-b359-c351cc5e8728 nodeName:}" failed. No retries permitted until 2026-09-08 18:42:47.340147204 +0000 UTC m=+1149.750457283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls") pod "maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn" (UID: "eadffdcc-faae-4890-b359-c351cc5e8728") : secret "maas-api-e2e-shared-b-w2-951c3e-serving-cert" not found
Sep 08 18:42:46.840315 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.840228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.840822 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.840800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadffdcc-faae-4890-b359-c351cc5e8728-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.842808 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.842785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:46.851981 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:46.851959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xz2t\" (UniqueName: \"kubernetes.io/projected/eadffdcc-faae-4890-b359-c351cc5e8728-kube-api-access-4xz2t\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:47.344224 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:47.344183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:47.346738 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:47.346709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:47.635408 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:47.635373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:47.919581 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:47.919458    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"]
Sep 08 18:42:47.922504 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:42:47.922476    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadffdcc_faae_4890_b359_c351cc5e8728.slice/crio-9df90f2cad337e9d20d2b044273c4a5ed7f29a736c4c0df0be65d71e12873d92 WatchSource:0}: Error finding container 9df90f2cad337e9d20d2b044273c4a5ed7f29a736c4c0df0be65d71e12873d92: Status 404 returned error can't find the container with id 9df90f2cad337e9d20d2b044273c4a5ed7f29a736c4c0df0be65d71e12873d92
Sep 08 18:42:48.498901 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:48.498853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn" event={"ID":"eadffdcc-faae-4890-b359-c351cc5e8728","Type":"ContainerStarted","Data":"b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554"}
Sep 08 18:42:48.499064 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:48.498958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:42:48.499064 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:48.498990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn" event={"ID":"eadffdcc-faae-4890-b359-c351cc5e8728","Type":"ContainerStarted","Data":"9df90f2cad337e9d20d2b044273c4a5ed7f29a736c4c0df0be65d71e12873d92"}
Sep 08 18:42:48.517617 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:48.517573    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn" podStartSLOduration=2.033894499 podStartE2EDuration="2.51755814s" totalImagesPullingTime="483.663641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:48.516025883 +0000 UTC m=+1150.926335997" watchObservedRunningTime="2026-09-08 18:42:48.51755814 +0000 UTC m=+1150.927868220"
Sep 08 18:42:54.509390 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:42:54.509357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:43:24.961845 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:24.961810    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"]
Sep 08 18:43:24.965761 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:24.965724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:24.969400 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:24.969375    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-783195cb-serving-cert\""
Sep 08 18:43:24.981754 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:24.981729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"]
Sep 08 18:43:25.010757 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.010720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-metrics-certs\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.010757 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.010757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjltv\" (UniqueName: \"kubernetes.io/projected/426a2a5f-5411-40c5-a4ad-8e2e2923b100-kube-api-access-rjltv\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.010952 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.010787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426a2a5f-5411-40c5-a4ad-8e2e2923b100-openshift-service-ca\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.010952 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.010879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.112326 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.112295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.112516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.112375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-metrics-certs\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.112516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.112396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjltv\" (UniqueName: \"kubernetes.io/projected/426a2a5f-5411-40c5-a4ad-8e2e2923b100-kube-api-access-rjltv\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.112516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.112454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426a2a5f-5411-40c5-a4ad-8e2e2923b100-openshift-service-ca\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.112516 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:43:25.112482    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-serving-cert: secret "maas-api-e2e-mt-783195cb-serving-cert" not found
Sep 08 18:43:25.112658 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:43:25.112569    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls podName:426a2a5f-5411-40c5-a4ad-8e2e2923b100 nodeName:}" failed. No retries permitted until 2026-09-08 18:43:25.61255101 +0000 UTC m=+1188.022861086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls") pod "maas-api-e2e-mt-783195cb-989754f59-9247c" (UID: "426a2a5f-5411-40c5-a4ad-8e2e2923b100") : secret "maas-api-e2e-mt-783195cb-serving-cert" not found
Sep 08 18:43:25.113116 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.113087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426a2a5f-5411-40c5-a4ad-8e2e2923b100-openshift-service-ca\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.114979 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.114960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-metrics-certs\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.121749 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.121728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjltv\" (UniqueName: \"kubernetes.io/projected/426a2a5f-5411-40c5-a4ad-8e2e2923b100-kube-api-access-rjltv\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.617502 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.617463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.620184 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.620159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls\") pod \"maas-api-e2e-mt-783195cb-989754f59-9247c\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:25.876467 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:25.876372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:26.158356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:26.158329    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"]
Sep 08 18:43:26.160313 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:43:26.160283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426a2a5f_5411_40c5_a4ad_8e2e2923b100.slice/crio-0c3f8eccccff91f0505c8a26f2be91f9f08a2f0081a672b56f65471f266750e9 WatchSource:0}: Error finding container 0c3f8eccccff91f0505c8a26f2be91f9f08a2f0081a672b56f65471f266750e9: Status 404 returned error can't find the container with id 0c3f8eccccff91f0505c8a26f2be91f9f08a2f0081a672b56f65471f266750e9
Sep 08 18:43:26.688573 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:26.688537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c" event={"ID":"426a2a5f-5411-40c5-a4ad-8e2e2923b100","Type":"ContainerStarted","Data":"0c3f8eccccff91f0505c8a26f2be91f9f08a2f0081a672b56f65471f266750e9"}
Sep 08 18:43:27.695782 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:27.695739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c" event={"ID":"426a2a5f-5411-40c5-a4ad-8e2e2923b100","Type":"ContainerStarted","Data":"bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288"}
Sep 08 18:43:27.696183 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:27.695866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:43:27.717502 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:27.717453    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c" podStartSLOduration=2.952331543 podStartE2EDuration="3.717436794s" totalImagesPullingTime="765.105251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:27.714894063 +0000 UTC m=+1190.125204143" watchObservedRunningTime="2026-09-08 18:43:27.717436794 +0000 UTC m=+1190.127746890"
Sep 08 18:43:33.706516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:43:33.706481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:44:33.473573 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.473403    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"]
Sep 08 18:44:33.476163 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.473815    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c" podUID="426a2a5f-5411-40c5-a4ad-8e2e2923b100" containerName="maas-api" containerID="cri-o://bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288" gracePeriod=30
Sep 08 18:44:33.750098 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.750070    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:44:33.827786 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.827749    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426a2a5f-5411-40c5-a4ad-8e2e2923b100-openshift-service-ca\") pod \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") "
Sep 08 18:44:33.827961 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.827802    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-metrics-certs\") pod \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") "
Sep 08 18:44:33.827961 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.827899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjltv\" (UniqueName: \"kubernetes.io/projected/426a2a5f-5411-40c5-a4ad-8e2e2923b100-kube-api-access-rjltv\") pod \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") "
Sep 08 18:44:33.827961 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.827938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls\") pod \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\" (UID: \"426a2a5f-5411-40c5-a4ad-8e2e2923b100\") "
Sep 08 18:44:33.828221 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.828194    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426a2a5f-5411-40c5-a4ad-8e2e2923b100-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "426a2a5f-5411-40c5-a4ad-8e2e2923b100" (UID: "426a2a5f-5411-40c5-a4ad-8e2e2923b100"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:44:33.830260 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.830229    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "426a2a5f-5411-40c5-a4ad-8e2e2923b100" (UID: "426a2a5f-5411-40c5-a4ad-8e2e2923b100"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:44:33.830260 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.830237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426a2a5f-5411-40c5-a4ad-8e2e2923b100-kube-api-access-rjltv" (OuterVolumeSpecName: "kube-api-access-rjltv") pod "426a2a5f-5411-40c5-a4ad-8e2e2923b100" (UID: "426a2a5f-5411-40c5-a4ad-8e2e2923b100"). InnerVolumeSpecName "kube-api-access-rjltv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:44:33.830260 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.830247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "426a2a5f-5411-40c5-a4ad-8e2e2923b100" (UID: "426a2a5f-5411-40c5-a4ad-8e2e2923b100"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:44:33.929603 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.929563    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjltv\" (UniqueName: \"kubernetes.io/projected/426a2a5f-5411-40c5-a4ad-8e2e2923b100-kube-api-access-rjltv\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:44:33.929790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.929615    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:44:33.929790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.929626    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/426a2a5f-5411-40c5-a4ad-8e2e2923b100-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:44:33.929790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:33.929634    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426a2a5f-5411-40c5-a4ad-8e2e2923b100-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:44:34.038485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.038363    2580 generic.go:356] "Generic (PLEG): container finished" podID="426a2a5f-5411-40c5-a4ad-8e2e2923b100" containerID="bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288" exitCode=0
Sep 08 18:44:34.038485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.038444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"
Sep 08 18:44:34.038485 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.038462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c" event={"ID":"426a2a5f-5411-40c5-a4ad-8e2e2923b100","Type":"ContainerDied","Data":"bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288"}
Sep 08 18:44:34.038711 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.038545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c" event={"ID":"426a2a5f-5411-40c5-a4ad-8e2e2923b100","Type":"ContainerDied","Data":"0c3f8eccccff91f0505c8a26f2be91f9f08a2f0081a672b56f65471f266750e9"}
Sep 08 18:44:34.038711 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.038570    2580 scope.go:117] "RemoveContainer" containerID="bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288"
Sep 08 18:44:34.048452 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.048414    2580 scope.go:117] "RemoveContainer" containerID="bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288"
Sep 08 18:44:34.048754 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:44:34.048734    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288\": container with ID starting with bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288 not found: ID does not exist" containerID="bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288"
Sep 08 18:44:34.048813 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.048765    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288"} err="failed to get container status \"bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288\": rpc error: code = NotFound desc = could not find container \"bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288\": container with ID starting with bb82299e02136a9ea9b1dc5318a63c7e1f50e07d0f31beaec46a028743167288 not found: ID does not exist"
Sep 08 18:44:34.061867 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.061840    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"]
Sep 08 18:44:34.067511 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.067488    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c"]
Sep 08 18:44:34.121875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:34.121832    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426a2a5f-5411-40c5-a4ad-8e2e2923b100" path="/var/lib/kubelet/pods/426a2a5f-5411-40c5-a4ad-8e2e2923b100/volumes"
Sep 08 18:44:36.701239 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:44:36.701194    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-783195cb-989754f59-9247c" podUID="426a2a5f-5411-40c5-a4ad-8e2e2923b100" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.52:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 08 18:46:29.129878 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.129843    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"]
Sep 08 18:46:29.130866 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.130843    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426a2a5f-5411-40c5-a4ad-8e2e2923b100" containerName="maas-api"
Sep 08 18:46:29.130866 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.130867    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="426a2a5f-5411-40c5-a4ad-8e2e2923b100" containerName="maas-api"
Sep 08 18:46:29.131006 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.130936    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="426a2a5f-5411-40c5-a4ad-8e2e2923b100" containerName="maas-api"
Sep 08 18:46:29.134105 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.134086    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.136677 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.136655    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ff349c-serving-cert\""
Sep 08 18:46:29.144933 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.144906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"]
Sep 08 18:46:29.225721 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.225689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddqr7\" (UniqueName: \"kubernetes.io/projected/a042281d-ee63-483e-a905-1496ba311913-kube-api-access-ddqr7\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.225906 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.225749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-maas-api-tls\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.225906 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.225806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a042281d-ee63-483e-a905-1496ba311913-openshift-service-ca\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.225986 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.225915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-metrics-certs\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.327046 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.327012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddqr7\" (UniqueName: \"kubernetes.io/projected/a042281d-ee63-483e-a905-1496ba311913-kube-api-access-ddqr7\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.327222 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.327062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-maas-api-tls\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.327222 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.327084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a042281d-ee63-483e-a905-1496ba311913-openshift-service-ca\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.327222 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.327171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-metrics-certs\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.327810 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.327787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a042281d-ee63-483e-a905-1496ba311913-openshift-service-ca\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.329689 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.329665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-maas-api-tls\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.329789 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.329751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-metrics-certs\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.336350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.336328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddqr7\" (UniqueName: \"kubernetes.io/projected/a042281d-ee63-483e-a905-1496ba311913-kube-api-access-ddqr7\") pod \"maas-api-e2e-api-a-ff349c-8db698f95-7jtbj\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.447024 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.446951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:29.728006 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.727979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"]
Sep 08 18:46:29.730342 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:46:29.730312    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda042281d_ee63_483e_a905_1496ba311913.slice/crio-d5d14c88c9e0a8bdc3bc4373702df68f9d6971d98c012cd78320f4d7001080d9 WatchSource:0}: Error finding container d5d14c88c9e0a8bdc3bc4373702df68f9d6971d98c012cd78320f4d7001080d9: Status 404 returned error can't find the container with id d5d14c88c9e0a8bdc3bc4373702df68f9d6971d98c012cd78320f4d7001080d9
Sep 08 18:46:29.731908 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:29.731887    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:46:30.613763 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:30.613726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj" event={"ID":"a042281d-ee63-483e-a905-1496ba311913","Type":"ContainerStarted","Data":"b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630"}
Sep 08 18:46:30.614146 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:30.613783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj" event={"ID":"a042281d-ee63-483e-a905-1496ba311913","Type":"ContainerStarted","Data":"d5d14c88c9e0a8bdc3bc4373702df68f9d6971d98c012cd78320f4d7001080d9"}
Sep 08 18:46:30.614146 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:30.613829    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:30.632892 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:30.632850    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj" podStartSLOduration=0.887701118 podStartE2EDuration="1.632835278s" totalImagesPullingTime="745.13416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:30.631756955 +0000 UTC m=+1373.042067036" watchObservedRunningTime="2026-09-08 18:46:30.632835278 +0000 UTC m=+1373.043145359"
Sep 08 18:46:36.624598 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:36.624568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:46:58.063353 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.063324    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"]
Sep 08 18:46:58.067414 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.067391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.070245 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.070223    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-179a6f-serving-cert\""
Sep 08 18:46:58.079512 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.079491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"]
Sep 08 18:46:58.213333 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.213296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-metrics-certs\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.213516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.213348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdcw\" (UniqueName: \"kubernetes.io/projected/15c11620-e117-4114-a6d4-f04dca880610-kube-api-access-gmdcw\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.213516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.213502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-maas-api-tls\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.213601 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.213531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c11620-e117-4114-a6d4-f04dca880610-openshift-service-ca\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.314699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.314592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-maas-api-tls\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.314699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.314633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c11620-e117-4114-a6d4-f04dca880610-openshift-service-ca\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.314699 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.314690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-metrics-certs\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.314992 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.314745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdcw\" (UniqueName: \"kubernetes.io/projected/15c11620-e117-4114-a6d4-f04dca880610-kube-api-access-gmdcw\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.315474 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.315419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c11620-e117-4114-a6d4-f04dca880610-openshift-service-ca\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.317398 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.317379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-maas-api-tls\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.317531 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.317396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-metrics-certs\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.323798 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.323774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdcw\" (UniqueName: \"kubernetes.io/projected/15c11620-e117-4114-a6d4-f04dca880610-kube-api-access-gmdcw\") pod \"maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.381361 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.381327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:58.657462 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.657417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"]
Sep 08 18:46:58.659678 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:46:58.659648    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c11620_e117_4114_a6d4_f04dca880610.slice/crio-1d627ea149db4578ca9c1d5efcd6b25ad538cedededf0642f9c5f1bf50f7b792 WatchSource:0}: Error finding container 1d627ea149db4578ca9c1d5efcd6b25ad538cedededf0642f9c5f1bf50f7b792: Status 404 returned error can't find the container with id 1d627ea149db4578ca9c1d5efcd6b25ad538cedededf0642f9c5f1bf50f7b792
Sep 08 18:46:58.753374 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:58.753339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd" event={"ID":"15c11620-e117-4114-a6d4-f04dca880610","Type":"ContainerStarted","Data":"1d627ea149db4578ca9c1d5efcd6b25ad538cedededf0642f9c5f1bf50f7b792"}
Sep 08 18:46:59.760044 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:59.760005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd" event={"ID":"15c11620-e117-4114-a6d4-f04dca880610","Type":"ContainerStarted","Data":"351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3"}
Sep 08 18:46:59.760541 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:59.760116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:46:59.780360 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:46:59.780307    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd" podStartSLOduration=1.153356272 podStartE2EDuration="1.780290565s" totalImagesPullingTime="626.934293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:59.778142607 +0000 UTC m=+1402.188452724" watchObservedRunningTime="2026-09-08 18:46:59.780290565 +0000 UTC m=+1402.190600646"
Sep 08 18:47:05.770242 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:05.770216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:47:40.894313 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:40.894230    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"]
Sep 08 18:47:40.894910 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:40.894547    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj" podUID="a042281d-ee63-483e-a905-1496ba311913" containerName="maas-api" containerID="cri-o://b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630" gracePeriod=30
Sep 08 18:47:41.154981 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.154923    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:47:41.233406 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.233376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddqr7\" (UniqueName: \"kubernetes.io/projected/a042281d-ee63-483e-a905-1496ba311913-kube-api-access-ddqr7\") pod \"a042281d-ee63-483e-a905-1496ba311913\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") "
Sep 08 18:47:41.233406 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.233412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a042281d-ee63-483e-a905-1496ba311913-openshift-service-ca\") pod \"a042281d-ee63-483e-a905-1496ba311913\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") "
Sep 08 18:47:41.233641 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.233474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-metrics-certs\") pod \"a042281d-ee63-483e-a905-1496ba311913\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") "
Sep 08 18:47:41.233641 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.233578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-maas-api-tls\") pod \"a042281d-ee63-483e-a905-1496ba311913\" (UID: \"a042281d-ee63-483e-a905-1496ba311913\") "
Sep 08 18:47:41.233849 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.233824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a042281d-ee63-483e-a905-1496ba311913-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a042281d-ee63-483e-a905-1496ba311913" (UID: "a042281d-ee63-483e-a905-1496ba311913"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:47:41.235720 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.235692    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a042281d-ee63-483e-a905-1496ba311913" (UID: "a042281d-ee63-483e-a905-1496ba311913"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:47:41.235841 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.235755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a042281d-ee63-483e-a905-1496ba311913-kube-api-access-ddqr7" (OuterVolumeSpecName: "kube-api-access-ddqr7") pod "a042281d-ee63-483e-a905-1496ba311913" (UID: "a042281d-ee63-483e-a905-1496ba311913"). InnerVolumeSpecName "kube-api-access-ddqr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:47:41.235907 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.235843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a042281d-ee63-483e-a905-1496ba311913" (UID: "a042281d-ee63-483e-a905-1496ba311913"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:47:41.334385 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.334347    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:47:41.334385 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.334378    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddqr7\" (UniqueName: \"kubernetes.io/projected/a042281d-ee63-483e-a905-1496ba311913-kube-api-access-ddqr7\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:47:41.334385 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.334389    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a042281d-ee63-483e-a905-1496ba311913-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:47:41.334641 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.334398    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a042281d-ee63-483e-a905-1496ba311913-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:47:41.970588 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.970558    2580 generic.go:356] "Generic (PLEG): container finished" podID="a042281d-ee63-483e-a905-1496ba311913" containerID="b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630" exitCode=0
Sep 08 18:47:41.971052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.970611    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"
Sep 08 18:47:41.971052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.970634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj" event={"ID":"a042281d-ee63-483e-a905-1496ba311913","Type":"ContainerDied","Data":"b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630"}
Sep 08 18:47:41.971052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.970701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj" event={"ID":"a042281d-ee63-483e-a905-1496ba311913","Type":"ContainerDied","Data":"d5d14c88c9e0a8bdc3bc4373702df68f9d6971d98c012cd78320f4d7001080d9"}
Sep 08 18:47:41.971052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.970736    2580 scope.go:117] "RemoveContainer" containerID="b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630"
Sep 08 18:47:41.980320 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.980301    2580 scope.go:117] "RemoveContainer" containerID="b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630"
Sep 08 18:47:41.980689 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:47:41.980664    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630\": container with ID starting with b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630 not found: ID does not exist" containerID="b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630"
Sep 08 18:47:41.980772 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.980700    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630"} err="failed to get container status \"b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630\": rpc error: code = NotFound desc = could not find container \"b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630\": container with ID starting with b8d2fe8ad945655434a579de85655f94cb946e473e29fb1aeb4e027589636630 not found: ID does not exist"
Sep 08 18:47:41.992859 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.992832    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"]
Sep 08 18:47:41.996965 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:41.996940    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ff349c-8db698f95-7jtbj"]
Sep 08 18:47:42.121046 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:47:42.121014    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a042281d-ee63-483e-a905-1496ba311913" path="/var/lib/kubelet/pods/a042281d-ee63-483e-a905-1496ba311913/volumes"
Sep 08 18:48:03.591476 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.591439    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"]
Sep 08 18:48:03.592441 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.592374    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd" podUID="15c11620-e117-4114-a6d4-f04dca880610" containerName="maas-api" containerID="cri-o://351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3" gracePeriod=30
Sep 08 18:48:03.849102 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.849045    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:48:03.957774 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.957743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-metrics-certs\") pod \"15c11620-e117-4114-a6d4-f04dca880610\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") "
Sep 08 18:48:03.957947 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.957814    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmdcw\" (UniqueName: \"kubernetes.io/projected/15c11620-e117-4114-a6d4-f04dca880610-kube-api-access-gmdcw\") pod \"15c11620-e117-4114-a6d4-f04dca880610\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") "
Sep 08 18:48:03.957947 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.957935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c11620-e117-4114-a6d4-f04dca880610-openshift-service-ca\") pod \"15c11620-e117-4114-a6d4-f04dca880610\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") "
Sep 08 18:48:03.958043 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.957985    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-maas-api-tls\") pod \"15c11620-e117-4114-a6d4-f04dca880610\" (UID: \"15c11620-e117-4114-a6d4-f04dca880610\") "
Sep 08 18:48:03.958380 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.958346    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c11620-e117-4114-a6d4-f04dca880610-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15c11620-e117-4114-a6d4-f04dca880610" (UID: "15c11620-e117-4114-a6d4-f04dca880610"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:48:03.960103 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.960078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c11620-e117-4114-a6d4-f04dca880610-kube-api-access-gmdcw" (OuterVolumeSpecName: "kube-api-access-gmdcw") pod "15c11620-e117-4114-a6d4-f04dca880610" (UID: "15c11620-e117-4114-a6d4-f04dca880610"). InnerVolumeSpecName "kube-api-access-gmdcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:03.960227 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.960204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15c11620-e117-4114-a6d4-f04dca880610" (UID: "15c11620-e117-4114-a6d4-f04dca880610"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:48:03.960646 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:03.960621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15c11620-e117-4114-a6d4-f04dca880610" (UID: "15c11620-e117-4114-a6d4-f04dca880610"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:48:04.059496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.059464    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c11620-e117-4114-a6d4-f04dca880610-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:04.059496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.059492    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:04.059496 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.059501    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15c11620-e117-4114-a6d4-f04dca880610-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:04.059746 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.059509    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmdcw\" (UniqueName: \"kubernetes.io/projected/15c11620-e117-4114-a6d4-f04dca880610-kube-api-access-gmdcw\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:04.082482 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.082418    2580 generic.go:356] "Generic (PLEG): container finished" podID="15c11620-e117-4114-a6d4-f04dca880610" containerID="351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3" exitCode=0
Sep 08 18:48:04.082482 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.082475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd" event={"ID":"15c11620-e117-4114-a6d4-f04dca880610","Type":"ContainerDied","Data":"351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3"}
Sep 08 18:48:04.082723 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.082538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd" event={"ID":"15c11620-e117-4114-a6d4-f04dca880610","Type":"ContainerDied","Data":"1d627ea149db4578ca9c1d5efcd6b25ad538cedededf0642f9c5f1bf50f7b792"}
Sep 08 18:48:04.082723 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.082487    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"
Sep 08 18:48:04.082723 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.082568    2580 scope.go:117] "RemoveContainer" containerID="351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3"
Sep 08 18:48:04.091812 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.091793    2580 scope.go:117] "RemoveContainer" containerID="351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3"
Sep 08 18:48:04.092095 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:48:04.092077    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3\": container with ID starting with 351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3 not found: ID does not exist" containerID="351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3"
Sep 08 18:48:04.092150 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.092103    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3"} err="failed to get container status \"351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3\": rpc error: code = NotFound desc = could not find container \"351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3\": container with ID starting with 351a1c418899c00cd83bd4e7abd664b9fee49e34182c968351f9807017441dc3 not found: ID does not exist"
Sep 08 18:48:04.103991 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.103934    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"]
Sep 08 18:48:04.107793 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.107773    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-179a6f-56bbc76689-bqcqd"]
Sep 08 18:48:04.120659 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:04.120638    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15c11620-e117-4114-a6d4-f04dca880610" path="/var/lib/kubelet/pods/15c11620-e117-4114-a6d4-f04dca880610/volumes"
Sep 08 18:48:21.102057 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.102025    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"]
Sep 08 18:48:21.102518 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.102247    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr" podUID="01719465-a7df-44de-ba36-6e170d86a468" containerName="maas-api" containerID="cri-o://5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80" gracePeriod=30
Sep 08 18:48:21.358718 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.358666    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:48:21.435201 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.435173    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-metrics-certs\") pod \"01719465-a7df-44de-ba36-6e170d86a468\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") "
Sep 08 18:48:21.435391 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.435212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5npw\" (UniqueName: \"kubernetes.io/projected/01719465-a7df-44de-ba36-6e170d86a468-kube-api-access-t5npw\") pod \"01719465-a7df-44de-ba36-6e170d86a468\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") "
Sep 08 18:48:21.435391 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.435242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01719465-a7df-44de-ba36-6e170d86a468-openshift-service-ca\") pod \"01719465-a7df-44de-ba36-6e170d86a468\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") "
Sep 08 18:48:21.435391 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.435352    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls\") pod \"01719465-a7df-44de-ba36-6e170d86a468\" (UID: \"01719465-a7df-44de-ba36-6e170d86a468\") "
Sep 08 18:48:21.435656 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.435634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01719465-a7df-44de-ba36-6e170d86a468-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01719465-a7df-44de-ba36-6e170d86a468" (UID: "01719465-a7df-44de-ba36-6e170d86a468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:48:21.437621 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.437512    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01719465-a7df-44de-ba36-6e170d86a468" (UID: "01719465-a7df-44de-ba36-6e170d86a468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:48:21.437621 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.437558    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01719465-a7df-44de-ba36-6e170d86a468-kube-api-access-t5npw" (OuterVolumeSpecName: "kube-api-access-t5npw") pod "01719465-a7df-44de-ba36-6e170d86a468" (UID: "01719465-a7df-44de-ba36-6e170d86a468"). InnerVolumeSpecName "kube-api-access-t5npw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:21.437621 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.437590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "01719465-a7df-44de-ba36-6e170d86a468" (UID: "01719465-a7df-44de-ba36-6e170d86a468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:48:21.536101 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.536065    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:21.536101 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.536091    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01719465-a7df-44de-ba36-6e170d86a468-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:21.536101 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.536102    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5npw\" (UniqueName: \"kubernetes.io/projected/01719465-a7df-44de-ba36-6e170d86a468-kube-api-access-t5npw\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:21.536101 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:21.536109    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01719465-a7df-44de-ba36-6e170d86a468-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:22.169973 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.169933    2580 generic.go:356] "Generic (PLEG): container finished" podID="01719465-a7df-44de-ba36-6e170d86a468" containerID="5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80" exitCode=0
Sep 08 18:48:22.170348 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.169997    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"
Sep 08 18:48:22.170348 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.170014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr" event={"ID":"01719465-a7df-44de-ba36-6e170d86a468","Type":"ContainerDied","Data":"5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80"}
Sep 08 18:48:22.170348 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.170084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr" event={"ID":"01719465-a7df-44de-ba36-6e170d86a468","Type":"ContainerDied","Data":"0aac0d977da128a602b80f0b76fdf7722ec3f66a7be4630557b07ea752731a96"}
Sep 08 18:48:22.170348 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.170108    2580 scope.go:117] "RemoveContainer" containerID="5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80"
Sep 08 18:48:22.180249 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.180225    2580 scope.go:117] "RemoveContainer" containerID="5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80"
Sep 08 18:48:22.180733 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:48:22.180714    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80\": container with ID starting with 5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80 not found: ID does not exist" containerID="5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80"
Sep 08 18:48:22.180814 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.180739    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80"} err="failed to get container status \"5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80\": rpc error: code = NotFound desc = could not find container \"5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80\": container with ID starting with 5dd388564b7cf6a84e6bcffa4ddd5d9f353a718a28f9ee1ef8d18dccafc14c80 not found: ID does not exist"
Sep 08 18:48:22.188348 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.188326    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"]
Sep 08 18:48:22.192254 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:22.192233    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3b04d6-76bb9445cd-297cr"]
Sep 08 18:48:24.119897 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:24.119865    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01719465-a7df-44de-ba36-6e170d86a468" path="/var/lib/kubelet/pods/01719465-a7df-44de-ba36-6e170d86a468/volumes"
Sep 08 18:48:38.185462 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.185357    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"]
Sep 08 18:48:38.186052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.185641    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn" podUID="eadffdcc-faae-4890-b359-c351cc5e8728" containerName="maas-api" containerID="cri-o://b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554" gracePeriod=30
Sep 08 18:48:38.438758 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.438689    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:48:38.608411 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.608378    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls\") pod \"eadffdcc-faae-4890-b359-c351cc5e8728\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") "
Sep 08 18:48:38.608596 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.608502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xz2t\" (UniqueName: \"kubernetes.io/projected/eadffdcc-faae-4890-b359-c351cc5e8728-kube-api-access-4xz2t\") pod \"eadffdcc-faae-4890-b359-c351cc5e8728\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") "
Sep 08 18:48:38.608596 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.608544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadffdcc-faae-4890-b359-c351cc5e8728-openshift-service-ca\") pod \"eadffdcc-faae-4890-b359-c351cc5e8728\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") "
Sep 08 18:48:38.608682 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.608608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-metrics-certs\") pod \"eadffdcc-faae-4890-b359-c351cc5e8728\" (UID: \"eadffdcc-faae-4890-b359-c351cc5e8728\") "
Sep 08 18:48:38.608998 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.608961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eadffdcc-faae-4890-b359-c351cc5e8728-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eadffdcc-faae-4890-b359-c351cc5e8728" (UID: "eadffdcc-faae-4890-b359-c351cc5e8728"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:48:38.610863 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.610821    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eadffdcc-faae-4890-b359-c351cc5e8728" (UID: "eadffdcc-faae-4890-b359-c351cc5e8728"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:48:38.610969 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.610860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadffdcc-faae-4890-b359-c351cc5e8728-kube-api-access-4xz2t" (OuterVolumeSpecName: "kube-api-access-4xz2t") pod "eadffdcc-faae-4890-b359-c351cc5e8728" (UID: "eadffdcc-faae-4890-b359-c351cc5e8728"). InnerVolumeSpecName "kube-api-access-4xz2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:38.610969 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.610902    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eadffdcc-faae-4890-b359-c351cc5e8728" (UID: "eadffdcc-faae-4890-b359-c351cc5e8728"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:48:38.709654 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.709562    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xz2t\" (UniqueName: \"kubernetes.io/projected/eadffdcc-faae-4890-b359-c351cc5e8728-kube-api-access-4xz2t\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:38.709654 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.709596    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadffdcc-faae-4890-b359-c351cc5e8728-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:38.709654 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.709610    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:38.709654 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:38.709621    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadffdcc-faae-4890-b359-c351cc5e8728-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:48:39.259091 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.259054    2580 generic.go:356] "Generic (PLEG): container finished" podID="eadffdcc-faae-4890-b359-c351cc5e8728" containerID="b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554" exitCode=0
Sep 08 18:48:39.259516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.259099    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"
Sep 08 18:48:39.259516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.259137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn" event={"ID":"eadffdcc-faae-4890-b359-c351cc5e8728","Type":"ContainerDied","Data":"b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554"}
Sep 08 18:48:39.259516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.259209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn" event={"ID":"eadffdcc-faae-4890-b359-c351cc5e8728","Type":"ContainerDied","Data":"9df90f2cad337e9d20d2b044273c4a5ed7f29a736c4c0df0be65d71e12873d92"}
Sep 08 18:48:39.259516 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.259243    2580 scope.go:117] "RemoveContainer" containerID="b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554"
Sep 08 18:48:39.268444 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.268409    2580 scope.go:117] "RemoveContainer" containerID="b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554"
Sep 08 18:48:39.268718 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:48:39.268699    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554\": container with ID starting with b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554 not found: ID does not exist" containerID="b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554"
Sep 08 18:48:39.268780 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.268730    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554"} err="failed to get container status \"b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554\": rpc error: code = NotFound desc = could not find container \"b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554\": container with ID starting with b5f9154687ce099e70d511c0c0532d8c7c6e3f009a90e1401cf0ede9edee6554 not found: ID does not exist"
Sep 08 18:48:39.279811 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.279788    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"]
Sep 08 18:48:39.283415 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:39.283390    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-951c3e-7b8f5ddf94-mc2jn"]
Sep 08 18:48:40.121133 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:48:40.121102    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eadffdcc-faae-4890-b359-c351cc5e8728" path="/var/lib/kubelet/pods/eadffdcc-faae-4890-b359-c351cc5e8728/volumes"
Sep 08 18:49:07.971353 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971245    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"]
Sep 08 18:49:07.971866 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971849    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01719465-a7df-44de-ba36-6e170d86a468" containerName="maas-api"
Sep 08 18:49:07.971918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971870    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01719465-a7df-44de-ba36-6e170d86a468" containerName="maas-api"
Sep 08 18:49:07.971918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971891    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c11620-e117-4114-a6d4-f04dca880610" containerName="maas-api"
Sep 08 18:49:07.971918 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971900    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c11620-e117-4114-a6d4-f04dca880610" containerName="maas-api"
Sep 08 18:49:07.972029 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971930    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a042281d-ee63-483e-a905-1496ba311913" containerName="maas-api"
Sep 08 18:49:07.972029 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971939    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a042281d-ee63-483e-a905-1496ba311913" containerName="maas-api"
Sep 08 18:49:07.972029 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eadffdcc-faae-4890-b359-c351cc5e8728" containerName="maas-api"
Sep 08 18:49:07.972029 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.971962    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eadffdcc-faae-4890-b359-c351cc5e8728" containerName="maas-api"
Sep 08 18:49:07.972170 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.972036    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c11620-e117-4114-a6d4-f04dca880610" containerName="maas-api"
Sep 08 18:49:07.972170 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.972048    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01719465-a7df-44de-ba36-6e170d86a468" containerName="maas-api"
Sep 08 18:49:07.972170 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.972058    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a042281d-ee63-483e-a905-1496ba311913" containerName="maas-api"
Sep 08 18:49:07.972170 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.972068    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eadffdcc-faae-4890-b359-c351cc5e8728" containerName="maas-api"
Sep 08 18:49:07.975360 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.975339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:07.978011 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.977988    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bc23ef-serving-cert\""
Sep 08 18:49:07.982264 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:07.982235    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"]
Sep 08 18:49:08.088837 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.088803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.089005 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.088884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.089005 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.088930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.089005 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.088954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfsqj\" (UniqueName: \"kubernetes.io/projected/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-kube-api-access-bfsqj\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.189462 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.189406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfsqj\" (UniqueName: \"kubernetes.io/projected/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-kube-api-access-bfsqj\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.189626 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.189502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.189626 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.189599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.189711 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:49:08.189647    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-serving-cert: secret "maas-api-e2e-ipp-a-bc23ef-serving-cert" not found
Sep 08 18:49:08.189711 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.189660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.189818 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:49:08.189715    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls podName:cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e nodeName:}" failed. No retries permitted until 2026-09-08 18:49:08.689699946 +0000 UTC m=+1531.100010006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls") pod "maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2" (UID: "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e") : secret "maas-api-e2e-ipp-a-bc23ef-serving-cert" not found
Sep 08 18:49:08.190221 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.190204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.192147 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.192126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.197413 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.197394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfsqj\" (UniqueName: \"kubernetes.io/projected/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-kube-api-access-bfsqj\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.694082 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.694050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.696615 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.696587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:08.889336 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:08.889298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:09.170523 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:09.170485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"]
Sep 08 18:49:09.171950 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:49:09.171922    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1ff0a3_720f_42b7_85e3_2210c6cb3f3e.slice/crio-c650094c064b8bcaecb383f1c5dd0e57cf7f132ce08ff444b81fb77e2025c649 WatchSource:0}: Error finding container c650094c064b8bcaecb383f1c5dd0e57cf7f132ce08ff444b81fb77e2025c649: Status 404 returned error can't find the container with id c650094c064b8bcaecb383f1c5dd0e57cf7f132ce08ff444b81fb77e2025c649
Sep 08 18:49:09.407097 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:09.407064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2" event={"ID":"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e","Type":"ContainerStarted","Data":"c650094c064b8bcaecb383f1c5dd0e57cf7f132ce08ff444b81fb77e2025c649"}
Sep 08 18:49:10.413306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:10.413271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2" event={"ID":"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e","Type":"ContainerStarted","Data":"786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab"}
Sep 08 18:49:10.413717 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:10.413336    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:10.431319 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:10.431270    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2" podStartSLOduration=2.721346145 podStartE2EDuration="3.431252361s" totalImagesPullingTime="709.906216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:10.429598843 +0000 UTC m=+1532.839908949" watchObservedRunningTime="2026-09-08 18:49:10.431252361 +0000 UTC m=+1532.841562441"
Sep 08 18:49:16.424661 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:16.424630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:49:37.119046 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.119009    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"]
Sep 08 18:49:37.122675 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.122652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.125076 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.125055    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-302afb-serving-cert\""
Sep 08 18:49:37.136235 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.136213    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"]
Sep 08 18:49:37.165201 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.165156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jpz\" (UniqueName: \"kubernetes.io/projected/7f15c3a5-eaf9-4389-a348-de76b5e76f19-kube-api-access-49jpz\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.165397 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.165286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.165397 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.165327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f15c3a5-eaf9-4389-a348-de76b5e76f19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.165397 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.165345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-metrics-certs\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.265943 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.265914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.265943 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.265948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f15c3a5-eaf9-4389-a348-de76b5e76f19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.266214 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.265968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-metrics-certs\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.266214 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.266060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jpz\" (UniqueName: \"kubernetes.io/projected/7f15c3a5-eaf9-4389-a348-de76b5e76f19-kube-api-access-49jpz\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.266214 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:49:37.266072    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-serving-cert: secret "maas-api-e2e-ipp-b-302afb-serving-cert" not found
Sep 08 18:49:37.266214 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:49:37.266138    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls podName:7f15c3a5-eaf9-4389-a348-de76b5e76f19 nodeName:}" failed. No retries permitted until 2026-09-08 18:49:37.766117606 +0000 UTC m=+1560.176427684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls") pod "maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc" (UID: "7f15c3a5-eaf9-4389-a348-de76b5e76f19") : secret "maas-api-e2e-ipp-b-302afb-serving-cert" not found
Sep 08 18:49:37.266732 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.266707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f15c3a5-eaf9-4389-a348-de76b5e76f19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.268739 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.268714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-metrics-certs\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.277805 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.277768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jpz\" (UniqueName: \"kubernetes.io/projected/7f15c3a5-eaf9-4389-a348-de76b5e76f19-kube-api-access-49jpz\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.771523 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.771486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:37.774102 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:37.774077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:38.034071 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:38.033987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:38.308191 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:38.308168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"]
Sep 08 18:49:38.310144 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:49:38.310108    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f15c3a5_eaf9_4389_a348_de76b5e76f19.slice/crio-8cd54e998be77900208967ae174f80f8b930c124f890f684264e36b4fcbffcbb WatchSource:0}: Error finding container 8cd54e998be77900208967ae174f80f8b930c124f890f684264e36b4fcbffcbb: Status 404 returned error can't find the container with id 8cd54e998be77900208967ae174f80f8b930c124f890f684264e36b4fcbffcbb
Sep 08 18:49:38.550738 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:38.550698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc" event={"ID":"7f15c3a5-eaf9-4389-a348-de76b5e76f19","Type":"ContainerStarted","Data":"8cd54e998be77900208967ae174f80f8b930c124f890f684264e36b4fcbffcbb"}
Sep 08 18:49:40.561155 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:40.561116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc" event={"ID":"7f15c3a5-eaf9-4389-a348-de76b5e76f19","Type":"ContainerStarted","Data":"40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111"}
Sep 08 18:49:40.561681 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:40.561249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:49:40.578929 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:40.578881    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc" podStartSLOduration=2.33156053 podStartE2EDuration="3.578866413s" totalImagesPullingTime="1.247305883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:40.577559344 +0000 UTC m=+1562.987869430" watchObservedRunningTime="2026-09-08 18:49:40.578866413 +0000 UTC m=+1562.989176498"
Sep 08 18:49:46.571844 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:49:46.571820    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:50:57.509867 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.509831    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"]
Sep 08 18:50:57.513668 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.513650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.517213 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.517194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-61f4b7-serving-cert\""
Sep 08 18:50:57.526422 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.526396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"]
Sep 08 18:50:57.633600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.633561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.633600 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.633610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5174bbd6-b935-43fa-bc2d-07e646bde8b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.633855 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.633643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.633855 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.633828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qb88\" (UniqueName: \"kubernetes.io/projected/5174bbd6-b935-43fa-bc2d-07e646bde8b2-kube-api-access-6qb88\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.735299 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.735258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qb88\" (UniqueName: \"kubernetes.io/projected/5174bbd6-b935-43fa-bc2d-07e646bde8b2-kube-api-access-6qb88\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.735513 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.735309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.735513 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.735327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5174bbd6-b935-43fa-bc2d-07e646bde8b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.735513 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.735351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.736118 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.736098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5174bbd6-b935-43fa-bc2d-07e646bde8b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.737943 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.737912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.738053 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.737982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.746321 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.746293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qb88\" (UniqueName: \"kubernetes.io/projected/5174bbd6-b935-43fa-bc2d-07e646bde8b2-kube-api-access-6qb88\") pod \"maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:57.824318 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:57.824212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:58.105566 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:58.105537    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"]
Sep 08 18:50:58.107932 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:50:58.107909    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5174bbd6_b935_43fa_bc2d_07e646bde8b2.slice/crio-ab3bf732c836be60423c6919db09253418a656c55914ccfd9989c5fb6f2458da WatchSource:0}: Error finding container ab3bf732c836be60423c6919db09253418a656c55914ccfd9989c5fb6f2458da: Status 404 returned error can't find the container with id ab3bf732c836be60423c6919db09253418a656c55914ccfd9989c5fb6f2458da
Sep 08 18:50:58.951861 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:58.951826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx" event={"ID":"5174bbd6-b935-43fa-bc2d-07e646bde8b2","Type":"ContainerStarted","Data":"ab3bf732c836be60423c6919db09253418a656c55914ccfd9989c5fb6f2458da"}
Sep 08 18:50:59.959251 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:59.959215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx" event={"ID":"5174bbd6-b935-43fa-bc2d-07e646bde8b2","Type":"ContainerStarted","Data":"1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab"}
Sep 08 18:50:59.959659 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:59.959319    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:50:59.983722 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:50:59.983669    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx" podStartSLOduration=2.066800762 podStartE2EDuration="2.983650042s" totalImagesPullingTime="916.84928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:50:59.9832902 +0000 UTC m=+1642.393600282" watchObservedRunningTime="2026-09-08 18:50:59.983650042 +0000 UTC m=+1642.393960123"
Sep 08 18:51:05.971469 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:05.971408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:51:28.094740 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.094706    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"]
Sep 08 18:51:28.095572 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.095515    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx" podUID="5174bbd6-b935-43fa-bc2d-07e646bde8b2" containerName="maas-api" containerID="cri-o://1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab" gracePeriod=30
Sep 08 18:51:28.358549 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.358486    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:51:28.432504 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.432470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-maas-api-tls\") pod \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") "
Sep 08 18:51:28.432670 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.432532    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5174bbd6-b935-43fa-bc2d-07e646bde8b2-openshift-service-ca\") pod \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") "
Sep 08 18:51:28.432670 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.432574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qb88\" (UniqueName: \"kubernetes.io/projected/5174bbd6-b935-43fa-bc2d-07e646bde8b2-kube-api-access-6qb88\") pod \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") "
Sep 08 18:51:28.432670 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.432653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-metrics-certs\") pod \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\" (UID: \"5174bbd6-b935-43fa-bc2d-07e646bde8b2\") "
Sep 08 18:51:28.432894 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.432871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5174bbd6-b935-43fa-bc2d-07e646bde8b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5174bbd6-b935-43fa-bc2d-07e646bde8b2" (UID: "5174bbd6-b935-43fa-bc2d-07e646bde8b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:51:28.434865 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.434838    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5174bbd6-b935-43fa-bc2d-07e646bde8b2" (UID: "5174bbd6-b935-43fa-bc2d-07e646bde8b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:51:28.434987 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.434870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5174bbd6-b935-43fa-bc2d-07e646bde8b2" (UID: "5174bbd6-b935-43fa-bc2d-07e646bde8b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:51:28.434987 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.434884    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5174bbd6-b935-43fa-bc2d-07e646bde8b2-kube-api-access-6qb88" (OuterVolumeSpecName: "kube-api-access-6qb88") pod "5174bbd6-b935-43fa-bc2d-07e646bde8b2" (UID: "5174bbd6-b935-43fa-bc2d-07e646bde8b2"). InnerVolumeSpecName "kube-api-access-6qb88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:51:28.533416 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.533388    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qb88\" (UniqueName: \"kubernetes.io/projected/5174bbd6-b935-43fa-bc2d-07e646bde8b2-kube-api-access-6qb88\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:51:28.533416 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.533413    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:51:28.533416 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.533422    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5174bbd6-b935-43fa-bc2d-07e646bde8b2-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:51:28.533775 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:28.533458    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5174bbd6-b935-43fa-bc2d-07e646bde8b2-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:51:29.108771 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.108730    2580 generic.go:356] "Generic (PLEG): container finished" podID="5174bbd6-b935-43fa-bc2d-07e646bde8b2" containerID="1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab" exitCode=0
Sep 08 18:51:29.109255 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.108799    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"
Sep 08 18:51:29.109255 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.108804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx" event={"ID":"5174bbd6-b935-43fa-bc2d-07e646bde8b2","Type":"ContainerDied","Data":"1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab"}
Sep 08 18:51:29.109255 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.108949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx" event={"ID":"5174bbd6-b935-43fa-bc2d-07e646bde8b2","Type":"ContainerDied","Data":"ab3bf732c836be60423c6919db09253418a656c55914ccfd9989c5fb6f2458da"}
Sep 08 18:51:29.109255 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.108972    2580 scope.go:117] "RemoveContainer" containerID="1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab"
Sep 08 18:51:29.118797 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.118778    2580 scope.go:117] "RemoveContainer" containerID="1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab"
Sep 08 18:51:29.119135 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:51:29.119115    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab\": container with ID starting with 1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab not found: ID does not exist" containerID="1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab"
Sep 08 18:51:29.119221 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.119143    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab"} err="failed to get container status \"1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab\": rpc error: code = NotFound desc = could not find container \"1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab\": container with ID starting with 1453aa630a84c663d6a42c5ad6721a9ada4b37e4fe97b66f7062edc292ddcaab not found: ID does not exist"
Sep 08 18:51:29.145168 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.145125    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"]
Sep 08 18:51:29.153065 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:29.153035    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-61f4b7-d85c8f64b-t79kx"]
Sep 08 18:51:30.119803 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:30.119764    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5174bbd6-b935-43fa-bc2d-07e646bde8b2" path="/var/lib/kubelet/pods/5174bbd6-b935-43fa-bc2d-07e646bde8b2/volumes"
Sep 08 18:51:54.954042 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:54.954006    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"]
Sep 08 18:51:54.954492 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:54.954200    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2" podUID="cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" containerName="maas-api" containerID="cri-o://786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab" gracePeriod=30
Sep 08 18:51:55.217186 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.217126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:51:55.237613 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.237586    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" containerID="786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab" exitCode=0
Sep 08 18:51:55.237767 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.237621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2" event={"ID":"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e","Type":"ContainerDied","Data":"786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab"}
Sep 08 18:51:55.237767 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.237625    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"
Sep 08 18:51:55.237767 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.237665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2" event={"ID":"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e","Type":"ContainerDied","Data":"c650094c064b8bcaecb383f1c5dd0e57cf7f132ce08ff444b81fb77e2025c649"}
Sep 08 18:51:55.237767 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.237689    2580 scope.go:117] "RemoveContainer" containerID="786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab"
Sep 08 18:51:55.246159 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.246144    2580 scope.go:117] "RemoveContainer" containerID="786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab"
Sep 08 18:51:55.246417 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:51:55.246395    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab\": container with ID starting with 786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab not found: ID does not exist" containerID="786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab"
Sep 08 18:51:55.246569 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.246441    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab"} err="failed to get container status \"786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab\": rpc error: code = NotFound desc = could not find container \"786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab\": container with ID starting with 786a863e286fd4da1b486ddf2aff3c0d1e031d0d702dacf5d14eec32d25b83ab not found: ID does not exist"
Sep 08 18:51:55.294726 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.294697    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-openshift-service-ca\") pod \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") "
Sep 08 18:51:55.294921 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.294760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-metrics-certs\") pod \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") "
Sep 08 18:51:55.294921 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.294830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls\") pod \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") "
Sep 08 18:51:55.294921 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.294900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfsqj\" (UniqueName: \"kubernetes.io/projected/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-kube-api-access-bfsqj\") pod \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\" (UID: \"cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e\") "
Sep 08 18:51:55.295114 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.295084    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" (UID: "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:51:55.297775 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.297732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" (UID: "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:51:55.297893 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.297828    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" (UID: "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:51:55.298262 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.298233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-kube-api-access-bfsqj" (OuterVolumeSpecName: "kube-api-access-bfsqj") pod "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" (UID: "cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e"). InnerVolumeSpecName "kube-api-access-bfsqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:51:55.396389 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.396357    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfsqj\" (UniqueName: \"kubernetes.io/projected/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-kube-api-access-bfsqj\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:51:55.396389 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.396381    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:51:55.396389 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.396390    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:51:55.396389 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.396398    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:51:55.575264 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.575235    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"]
Sep 08 18:51:55.579266 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:55.579244    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bc23ef-84fd894c76-c2fn2"]
Sep 08 18:51:56.121103 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:51:56.121071    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" path="/var/lib/kubelet/pods/cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e/volumes"
Sep 08 18:53:09.326026 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.325990    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"]
Sep 08 18:53:09.326876 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.326845    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc" podUID="7f15c3a5-eaf9-4389-a348-de76b5e76f19" containerName="maas-api" containerID="cri-o://40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111" gracePeriod=30
Sep 08 18:53:09.587408 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.587354    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:53:09.607879 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.607847    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f15c3a5-eaf9-4389-a348-de76b5e76f19" containerID="40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111" exitCode=0
Sep 08 18:53:09.608038 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.607885    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"
Sep 08 18:53:09.608038 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.607923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc" event={"ID":"7f15c3a5-eaf9-4389-a348-de76b5e76f19","Type":"ContainerDied","Data":"40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111"}
Sep 08 18:53:09.608038 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.607982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc" event={"ID":"7f15c3a5-eaf9-4389-a348-de76b5e76f19","Type":"ContainerDied","Data":"8cd54e998be77900208967ae174f80f8b930c124f890f684264e36b4fcbffcbb"}
Sep 08 18:53:09.608038 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.608003    2580 scope.go:117] "RemoveContainer" containerID="40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111"
Sep 08 18:53:09.626589 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.626569    2580 scope.go:117] "RemoveContainer" containerID="40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111"
Sep 08 18:53:09.626962 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:53:09.626935    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111\": container with ID starting with 40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111 not found: ID does not exist" containerID="40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111"
Sep 08 18:53:09.627068 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.626975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111"} err="failed to get container status \"40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111\": rpc error: code = NotFound desc = could not find container \"40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111\": container with ID starting with 40b7f0f8dd6fea57eff25e1bfcce5f0d883dba8b62f9d42a15c2c35ecafe9111 not found: ID does not exist"
Sep 08 18:53:09.705239 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.705212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49jpz\" (UniqueName: \"kubernetes.io/projected/7f15c3a5-eaf9-4389-a348-de76b5e76f19-kube-api-access-49jpz\") pod \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") "
Sep 08 18:53:09.705413 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.705292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-metrics-certs\") pod \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") "
Sep 08 18:53:09.705413 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.705321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f15c3a5-eaf9-4389-a348-de76b5e76f19-openshift-service-ca\") pod \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") "
Sep 08 18:53:09.705413 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.705353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls\") pod \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\" (UID: \"7f15c3a5-eaf9-4389-a348-de76b5e76f19\") "
Sep 08 18:53:09.705729 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.705698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f15c3a5-eaf9-4389-a348-de76b5e76f19-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f15c3a5-eaf9-4389-a348-de76b5e76f19" (UID: "7f15c3a5-eaf9-4389-a348-de76b5e76f19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:09.707468 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.707420    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f15c3a5-eaf9-4389-a348-de76b5e76f19-kube-api-access-49jpz" (OuterVolumeSpecName: "kube-api-access-49jpz") pod "7f15c3a5-eaf9-4389-a348-de76b5e76f19" (UID: "7f15c3a5-eaf9-4389-a348-de76b5e76f19"). InnerVolumeSpecName "kube-api-access-49jpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:09.707468 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.707454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7f15c3a5-eaf9-4389-a348-de76b5e76f19" (UID: "7f15c3a5-eaf9-4389-a348-de76b5e76f19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:09.707622 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.707606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f15c3a5-eaf9-4389-a348-de76b5e76f19" (UID: "7f15c3a5-eaf9-4389-a348-de76b5e76f19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:09.806306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.806272    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:09.806306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.806304    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f15c3a5-eaf9-4389-a348-de76b5e76f19-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:09.806306 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.806313    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f15c3a5-eaf9-4389-a348-de76b5e76f19-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:09.806580 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.806322    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jpz\" (UniqueName: \"kubernetes.io/projected/7f15c3a5-eaf9-4389-a348-de76b5e76f19-kube-api-access-49jpz\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:09.935152 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.935120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"]
Sep 08 18:53:09.941532 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:09.941505    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-302afb-5dbf9f6bc8-q7zzc"]
Sep 08 18:53:10.120685 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:10.120658    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f15c3a5-eaf9-4389-a348-de76b5e76f19" path="/var/lib/kubelet/pods/7f15c3a5-eaf9-4389-a348-de76b5e76f19/volumes"
Sep 08 18:53:26.709979 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:26.709944    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"]
Sep 08 18:53:26.711223 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:26.711174    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m" podUID="e827d117-24c2-4d47-b054-b202f764b498" containerName="maas-api" containerID="cri-o://1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302" gracePeriod=30
Sep 08 18:53:26.976062 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:26.975999    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:53:27.071672 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.071641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls\") pod \"e827d117-24c2-4d47-b054-b202f764b498\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") "
Sep 08 18:53:27.071887 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.071718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-metrics-certs\") pod \"e827d117-24c2-4d47-b054-b202f764b498\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") "
Sep 08 18:53:27.071887 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.071772    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzzhh\" (UniqueName: \"kubernetes.io/projected/e827d117-24c2-4d47-b054-b202f764b498-kube-api-access-pzzhh\") pod \"e827d117-24c2-4d47-b054-b202f764b498\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") "
Sep 08 18:53:27.071887 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.071823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e827d117-24c2-4d47-b054-b202f764b498-openshift-service-ca\") pod \"e827d117-24c2-4d47-b054-b202f764b498\" (UID: \"e827d117-24c2-4d47-b054-b202f764b498\") "
Sep 08 18:53:27.072202 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.072165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e827d117-24c2-4d47-b054-b202f764b498-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e827d117-24c2-4d47-b054-b202f764b498" (UID: "e827d117-24c2-4d47-b054-b202f764b498"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:27.074176 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.074093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e827d117-24c2-4d47-b054-b202f764b498" (UID: "e827d117-24c2-4d47-b054-b202f764b498"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:27.074733 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.074178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e827d117-24c2-4d47-b054-b202f764b498-kube-api-access-pzzhh" (OuterVolumeSpecName: "kube-api-access-pzzhh") pod "e827d117-24c2-4d47-b054-b202f764b498" (UID: "e827d117-24c2-4d47-b054-b202f764b498"). InnerVolumeSpecName "kube-api-access-pzzhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:27.074733 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.074361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e827d117-24c2-4d47-b054-b202f764b498" (UID: "e827d117-24c2-4d47-b054-b202f764b498"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:27.173012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.172965    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e827d117-24c2-4d47-b054-b202f764b498-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:27.173012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.173008    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:27.173012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.173018    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e827d117-24c2-4d47-b054-b202f764b498-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:27.173012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.173026    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzzhh\" (UniqueName: \"kubernetes.io/projected/e827d117-24c2-4d47-b054-b202f764b498-kube-api-access-pzzhh\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:27.703483 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.703454    2580 generic.go:356] "Generic (PLEG): container finished" podID="e827d117-24c2-4d47-b054-b202f764b498" containerID="1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302" exitCode=0
Sep 08 18:53:27.703667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.703560    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"
Sep 08 18:53:27.703667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.703580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m" event={"ID":"e827d117-24c2-4d47-b054-b202f764b498","Type":"ContainerDied","Data":"1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302"}
Sep 08 18:53:27.703667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.703620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m" event={"ID":"e827d117-24c2-4d47-b054-b202f764b498","Type":"ContainerDied","Data":"fe9e81a4ff714914b0a389c76a3750b926774ee59541a5df9c2610dd23dd5635"}
Sep 08 18:53:27.703667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.703641    2580 scope.go:117] "RemoveContainer" containerID="1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302"
Sep 08 18:53:27.713012 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.712989    2580 scope.go:117] "RemoveContainer" containerID="1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302"
Sep 08 18:53:27.713289 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:53:27.713273    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302\": container with ID starting with 1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302 not found: ID does not exist" containerID="1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302"
Sep 08 18:53:27.713330 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.713302    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302"} err="failed to get container status \"1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302\": rpc error: code = NotFound desc = could not find container \"1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302\": container with ID starting with 1e463e27c5f9498d30fcc3a5358c3e6c5eb45d79a6db7ddc57d0404ead614302 not found: ID does not exist"
Sep 08 18:53:27.724712 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.724682    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"]
Sep 08 18:53:27.727875 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:27.727852    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-792c35-7658dcd5cc-bfr8m"]
Sep 08 18:53:28.120269 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:28.120239    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e827d117-24c2-4d47-b054-b202f764b498" path="/var/lib/kubelet/pods/e827d117-24c2-4d47-b054-b202f764b498/volumes"
Sep 08 18:53:44.590705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.590268    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"]
Sep 08 18:53:44.590705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.590537    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r" podUID="2ac8d07e-7e9a-4596-a151-748fda456e9f" containerName="maas-api" containerID="cri-o://582f97530b27fb97a3c7fff90a1f5e4777ba86b8ee1914a3d213fc692bcbdf80" gracePeriod=30
Sep 08 18:53:44.792252 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.792198    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac8d07e-7e9a-4596-a151-748fda456e9f" containerID="582f97530b27fb97a3c7fff90a1f5e4777ba86b8ee1914a3d213fc692bcbdf80" exitCode=0
Sep 08 18:53:44.792514 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.792361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r" event={"ID":"2ac8d07e-7e9a-4596-a151-748fda456e9f","Type":"ContainerDied","Data":"582f97530b27fb97a3c7fff90a1f5e4777ba86b8ee1914a3d213fc692bcbdf80"}
Sep 08 18:53:44.850887 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.850824    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:53:44.941180 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.941151    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-metrics-certs\") pod \"2ac8d07e-7e9a-4596-a151-748fda456e9f\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") "
Sep 08 18:53:44.941361 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.941227    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls\") pod \"2ac8d07e-7e9a-4596-a151-748fda456e9f\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") "
Sep 08 18:53:44.941361 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.941270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fslhz\" (UniqueName: \"kubernetes.io/projected/2ac8d07e-7e9a-4596-a151-748fda456e9f-kube-api-access-fslhz\") pod \"2ac8d07e-7e9a-4596-a151-748fda456e9f\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") "
Sep 08 18:53:44.941361 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.941350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac8d07e-7e9a-4596-a151-748fda456e9f-openshift-service-ca\") pod \"2ac8d07e-7e9a-4596-a151-748fda456e9f\" (UID: \"2ac8d07e-7e9a-4596-a151-748fda456e9f\") "
Sep 08 18:53:44.941731 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.941705    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac8d07e-7e9a-4596-a151-748fda456e9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ac8d07e-7e9a-4596-a151-748fda456e9f" (UID: "2ac8d07e-7e9a-4596-a151-748fda456e9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:44.943900 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.943849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ac8d07e-7e9a-4596-a151-748fda456e9f" (UID: "2ac8d07e-7e9a-4596-a151-748fda456e9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:44.944021 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.943941    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac8d07e-7e9a-4596-a151-748fda456e9f-kube-api-access-fslhz" (OuterVolumeSpecName: "kube-api-access-fslhz") pod "2ac8d07e-7e9a-4596-a151-748fda456e9f" (UID: "2ac8d07e-7e9a-4596-a151-748fda456e9f"). InnerVolumeSpecName "kube-api-access-fslhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:44.944021 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:44.943957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ac8d07e-7e9a-4596-a151-748fda456e9f" (UID: "2ac8d07e-7e9a-4596-a151-748fda456e9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:45.042305 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.042267    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:45.042305 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.042299    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac8d07e-7e9a-4596-a151-748fda456e9f-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:45.042305 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.042315    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fslhz\" (UniqueName: \"kubernetes.io/projected/2ac8d07e-7e9a-4596-a151-748fda456e9f-kube-api-access-fslhz\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:45.042571 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.042325    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac8d07e-7e9a-4596-a151-748fda456e9f-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:45.798179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.798143    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"
Sep 08 18:53:45.798179 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.798170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r" event={"ID":"2ac8d07e-7e9a-4596-a151-748fda456e9f","Type":"ContainerDied","Data":"9e3a9023d5f8f15db1ad5944287b03920d1c25d5be658a5c0655f8ca27e0ccbb"}
Sep 08 18:53:45.798721 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.798234    2580 scope.go:117] "RemoveContainer" containerID="582f97530b27fb97a3c7fff90a1f5e4777ba86b8ee1914a3d213fc692bcbdf80"
Sep 08 18:53:45.820164 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.820133    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"]
Sep 08 18:53:45.823815 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:45.823792    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6255f2-6dc478f55c-2lk2r"]
Sep 08 18:53:46.120154 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:46.120122    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac8d07e-7e9a-4596-a151-748fda456e9f" path="/var/lib/kubelet/pods/2ac8d07e-7e9a-4596-a151-748fda456e9f/volumes"
Sep 08 18:53:53.879976 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:53.879935    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-l6c7j"]
Sep 08 18:53:53.880554 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:53.880210    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j" podUID="102a765e-9699-40c7-a763-83d46e5b89ff" containerName="manager" containerID="cri-o://1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850" gracePeriod=30
Sep 08 18:53:54.137894 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.137837    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:53:54.327005 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.326973    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-metrics-certs\") pod \"102a765e-9699-40c7-a763-83d46e5b89ff\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") "
Sep 08 18:53:54.327185 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.327017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-webhook-cert\") pod \"102a765e-9699-40c7-a763-83d46e5b89ff\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") "
Sep 08 18:53:54.327185 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.327054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s89mm\" (UniqueName: \"kubernetes.io/projected/102a765e-9699-40c7-a763-83d46e5b89ff-kube-api-access-s89mm\") pod \"102a765e-9699-40c7-a763-83d46e5b89ff\" (UID: \"102a765e-9699-40c7-a763-83d46e5b89ff\") "
Sep 08 18:53:54.329323 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.329296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "102a765e-9699-40c7-a763-83d46e5b89ff" (UID: "102a765e-9699-40c7-a763-83d46e5b89ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:54.329456 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.329352    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102a765e-9699-40c7-a763-83d46e5b89ff-kube-api-access-s89mm" (OuterVolumeSpecName: "kube-api-access-s89mm") pod "102a765e-9699-40c7-a763-83d46e5b89ff" (UID: "102a765e-9699-40c7-a763-83d46e5b89ff"). InnerVolumeSpecName "kube-api-access-s89mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:54.329456 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.329376    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "102a765e-9699-40c7-a763-83d46e5b89ff" (UID: "102a765e-9699-40c7-a763-83d46e5b89ff"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:53:54.428709 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.428631    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-webhook-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.428709 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.428658    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s89mm\" (UniqueName: \"kubernetes.io/projected/102a765e-9699-40c7-a763-83d46e5b89ff-kube-api-access-s89mm\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.428709 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.428668    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a765e-9699-40c7-a763-83d46e5b89ff-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:53:54.839667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.839577    2580 generic.go:356] "Generic (PLEG): container finished" podID="102a765e-9699-40c7-a763-83d46e5b89ff" containerID="1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850" exitCode=0
Sep 08 18:53:54.839667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.839628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j" event={"ID":"102a765e-9699-40c7-a763-83d46e5b89ff","Type":"ContainerDied","Data":"1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850"}
Sep 08 18:53:54.839667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.839629    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j"
Sep 08 18:53:54.839667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.839669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-l6c7j" event={"ID":"102a765e-9699-40c7-a763-83d46e5b89ff","Type":"ContainerDied","Data":"9abde93a5e12046b8c2cac6f5b5b59f80129ee7f606c34bffa7702647ab2b0e7"}
Sep 08 18:53:54.839997 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.839690    2580 scope.go:117] "RemoveContainer" containerID="1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850"
Sep 08 18:53:54.848981 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.848964    2580 scope.go:117] "RemoveContainer" containerID="1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850"
Sep 08 18:53:54.849277 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:53:54.849259    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850\": container with ID starting with 1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850 not found: ID does not exist" containerID="1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850"
Sep 08 18:53:54.849328 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.849286    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850"} err="failed to get container status \"1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850\": rpc error: code = NotFound desc = could not find container \"1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850\": container with ID starting with 1ad3b36dee1269df71305b9fadf5dc2f0aa9d0164937051d50905fbbf568c850 not found: ID does not exist"
Sep 08 18:53:54.860590 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.860540    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-l6c7j"]
Sep 08 18:53:54.862138 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:54.862120    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-l6c7j"]
Sep 08 18:53:56.120049 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:53:56.120013    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102a765e-9699-40c7-a763-83d46e5b89ff" path="/var/lib/kubelet/pods/102a765e-9699-40c7-a763-83d46e5b89ff/volumes"
Sep 08 18:54:55.655474 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655437    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-d4zbk"]
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655781    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e827d117-24c2-4d47-b054-b202f764b498" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655791    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e827d117-24c2-4d47-b054-b202f764b498" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655798    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f15c3a5-eaf9-4389-a348-de76b5e76f19" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655803    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f15c3a5-eaf9-4389-a348-de76b5e76f19" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655809    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102a765e-9699-40c7-a763-83d46e5b89ff" containerName="manager"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655815    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="102a765e-9699-40c7-a763-83d46e5b89ff" containerName="manager"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655823    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5174bbd6-b935-43fa-bc2d-07e646bde8b2" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655828    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5174bbd6-b935-43fa-bc2d-07e646bde8b2" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655844    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655853    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac8d07e-7e9a-4596-a151-748fda456e9f" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655858    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac8d07e-7e9a-4596-a151-748fda456e9f" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655912    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac8d07e-7e9a-4596-a151-748fda456e9f" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655918    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5174bbd6-b935-43fa-bc2d-07e646bde8b2" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655924    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc1ff0a3-720f-42b7-85e3-2210c6cb3f3e" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655929    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f15c3a5-eaf9-4389-a348-de76b5e76f19" containerName="maas-api"
Sep 08 18:54:55.655932 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655939    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="102a765e-9699-40c7-a763-83d46e5b89ff" containerName="manager"
Sep 08 18:54:55.656596 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.655945    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e827d117-24c2-4d47-b054-b202f764b498" containerName="maas-api"
Sep 08 18:54:55.659040 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.659021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.663343 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.663304    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 18:54:55.663343 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.663318    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r7hq9\""
Sep 08 18:54:55.663533 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.663391    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 18:54:55.663533 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.663416    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 18:54:55.668093 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.668067    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-d4zbk"]
Sep 08 18:54:55.770667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.770629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-metrics-certs\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.770667 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.770672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-webhook-cert\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.770869 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.770748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjn7n\" (UniqueName: \"kubernetes.io/projected/4aea8387-8549-40b2-b3ad-b1673660f0d0-kube-api-access-zjn7n\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.871258 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.871225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-metrics-certs\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.871455 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.871262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-webhook-cert\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.871455 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.871315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjn7n\" (UniqueName: \"kubernetes.io/projected/4aea8387-8549-40b2-b3ad-b1673660f0d0-kube-api-access-zjn7n\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.873978 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.873955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-webhook-cert\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.874058 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.873984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-metrics-certs\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.880546 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.880518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjn7n\" (UniqueName: \"kubernetes.io/projected/4aea8387-8549-40b2-b3ad-b1673660f0d0-kube-api-access-zjn7n\") pod \"maas-controller-865fc4c5d4-d4zbk\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") " pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:55.970062 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:55.969976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:56.245203 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:56.245178    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-d4zbk"]
Sep 08 18:54:56.247729 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:54:56.247694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aea8387_8549_40b2_b3ad_b1673660f0d0.slice/crio-5cb4bbe161cd0268e6185458233e2c9644a1d754802454e2b5cd19684c9d0b6f WatchSource:0}: Error finding container 5cb4bbe161cd0268e6185458233e2c9644a1d754802454e2b5cd19684c9d0b6f: Status 404 returned error can't find the container with id 5cb4bbe161cd0268e6185458233e2c9644a1d754802454e2b5cd19684c9d0b6f
Sep 08 18:54:56.249090 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:56.249070    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:54:57.135390 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:57.135352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk" event={"ID":"4aea8387-8549-40b2-b3ad-b1673660f0d0","Type":"ContainerStarted","Data":"4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241"}
Sep 08 18:54:57.135843 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:57.135460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk" event={"ID":"4aea8387-8549-40b2-b3ad-b1673660f0d0","Type":"ContainerStarted","Data":"5cb4bbe161cd0268e6185458233e2c9644a1d754802454e2b5cd19684c9d0b6f"}
Sep 08 18:54:57.135843 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:57.135511    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:54:57.154348 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:54:57.154299    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk" podStartSLOduration=1.5606397140000001 podStartE2EDuration="2.154283159s" totalImagesPullingTime="593.643445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:57.152682189 +0000 UTC m=+1879.562992294" watchObservedRunningTime="2026-09-08 18:54:57.154283159 +0000 UTC m=+1879.564593240"
Sep 08 18:55:08.142964 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:08.142889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:55:17.531631 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.531574    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-d4zbk"]
Sep 08 18:55:17.532169 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.532123    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk" podUID="4aea8387-8549-40b2-b3ad-b1673660f0d0" containerName="manager" containerID="cri-o://4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241" gracePeriod=30
Sep 08 18:55:17.776537 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.776508    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:55:17.879751 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.879726    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjn7n\" (UniqueName: \"kubernetes.io/projected/4aea8387-8549-40b2-b3ad-b1673660f0d0-kube-api-access-zjn7n\") pod \"4aea8387-8549-40b2-b3ad-b1673660f0d0\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") "
Sep 08 18:55:17.879921 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.879801    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-webhook-cert\") pod \"4aea8387-8549-40b2-b3ad-b1673660f0d0\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") "
Sep 08 18:55:17.879921 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.879831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-metrics-certs\") pod \"4aea8387-8549-40b2-b3ad-b1673660f0d0\" (UID: \"4aea8387-8549-40b2-b3ad-b1673660f0d0\") "
Sep 08 18:55:17.882138 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.882084    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aea8387-8549-40b2-b3ad-b1673660f0d0-kube-api-access-zjn7n" (OuterVolumeSpecName: "kube-api-access-zjn7n") pod "4aea8387-8549-40b2-b3ad-b1673660f0d0" (UID: "4aea8387-8549-40b2-b3ad-b1673660f0d0"). InnerVolumeSpecName "kube-api-access-zjn7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:55:17.882247 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.882127    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4aea8387-8549-40b2-b3ad-b1673660f0d0" (UID: "4aea8387-8549-40b2-b3ad-b1673660f0d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:55:17.882247 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.882142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4aea8387-8549-40b2-b3ad-b1673660f0d0" (UID: "4aea8387-8549-40b2-b3ad-b1673660f0d0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:55:17.981503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.981469    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-webhook-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:55:17.981503 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.981505    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aea8387-8549-40b2-b3ad-b1673660f0d0-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:55:17.981705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:17.981523    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjn7n\" (UniqueName: \"kubernetes.io/projected/4aea8387-8549-40b2-b3ad-b1673660f0d0-kube-api-access-zjn7n\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:55:18.233669 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.233563    2580 generic.go:356] "Generic (PLEG): container finished" podID="4aea8387-8549-40b2-b3ad-b1673660f0d0" containerID="4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241" exitCode=0
Sep 08 18:55:18.233669 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.233617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk" event={"ID":"4aea8387-8549-40b2-b3ad-b1673660f0d0","Type":"ContainerDied","Data":"4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241"}
Sep 08 18:55:18.233915 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.233673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk" event={"ID":"4aea8387-8549-40b2-b3ad-b1673660f0d0","Type":"ContainerDied","Data":"5cb4bbe161cd0268e6185458233e2c9644a1d754802454e2b5cd19684c9d0b6f"}
Sep 08 18:55:18.233915 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.233699    2580 scope.go:117] "RemoveContainer" containerID="4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241"
Sep 08 18:55:18.233915 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.233624    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-d4zbk"
Sep 08 18:55:18.242493 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.242475    2580 scope.go:117] "RemoveContainer" containerID="4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241"
Sep 08 18:55:18.242789 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:55:18.242771    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241\": container with ID starting with 4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241 not found: ID does not exist" containerID="4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241"
Sep 08 18:55:18.242836 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.242799    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241"} err="failed to get container status \"4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241\": rpc error: code = NotFound desc = could not find container \"4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241\": container with ID starting with 4b47243100f405a897c0bc201cffedfc2ae2efbb8a175ec9a7d5111c7b6ec241 not found: ID does not exist"
Sep 08 18:55:18.251039 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.251016    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-d4zbk"]
Sep 08 18:55:18.255531 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:18.255507    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-d4zbk"]
Sep 08 18:55:19.063140 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.063109    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-dwpl6"]
Sep 08 18:55:19.063530 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.063475    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aea8387-8549-40b2-b3ad-b1673660f0d0" containerName="manager"
Sep 08 18:55:19.063530 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.063487    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aea8387-8549-40b2-b3ad-b1673660f0d0" containerName="manager"
Sep 08 18:55:19.063610 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.063544    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aea8387-8549-40b2-b3ad-b1673660f0d0" containerName="manager"
Sep 08 18:55:19.067790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.067766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.070598 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.070576    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 18:55:19.071840 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.071795    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 18:55:19.071957 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.071880    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r7hq9\""
Sep 08 18:55:19.072548 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.072527    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 18:55:19.078209 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.078189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-dwpl6"]
Sep 08 18:55:19.192512 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.192476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-metrics-certs\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.192674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.192521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdjm\" (UniqueName: \"kubernetes.io/projected/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-kube-api-access-fmdjm\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.192674 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.192545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-webhook-cert\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.294090 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.294050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdjm\" (UniqueName: \"kubernetes.io/projected/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-kube-api-access-fmdjm\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.294293 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.294097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-webhook-cert\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.294293 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.294229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-metrics-certs\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.296790 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.296760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-webhook-cert\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.296902 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.296805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-metrics-certs\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.302928 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.302907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdjm\" (UniqueName: \"kubernetes.io/projected/7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4-kube-api-access-fmdjm\") pod \"maas-controller-865fc4c5d4-dwpl6\" (UID: \"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4\") " pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.381248 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.381216    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:19.652314 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:19.652273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-865fc4c5d4-dwpl6"]
Sep 08 18:55:19.654687 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:55:19.654653    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd29fa6_0b1d_4e07_a0e0_746d0b5c4ae4.slice/crio-2a41859498f4138059f6c72d86c5b59a5106e408ec5a0ae30869c3b9688ca0a6 WatchSource:0}: Error finding container 2a41859498f4138059f6c72d86c5b59a5106e408ec5a0ae30869c3b9688ca0a6: Status 404 returned error can't find the container with id 2a41859498f4138059f6c72d86c5b59a5106e408ec5a0ae30869c3b9688ca0a6
Sep 08 18:55:20.120151 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:20.120116    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aea8387-8549-40b2-b3ad-b1673660f0d0" path="/var/lib/kubelet/pods/4aea8387-8549-40b2-b3ad-b1673660f0d0/volumes"
Sep 08 18:55:20.246652 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:20.246620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-dwpl6" event={"ID":"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4","Type":"ContainerStarted","Data":"0ce397e9d66f55b28aee32df886e1fc8cf47d75c8d081d913bb65342b589a0e9"}
Sep 08 18:55:20.246840 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:20.246674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865fc4c5d4-dwpl6" event={"ID":"7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4","Type":"ContainerStarted","Data":"2a41859498f4138059f6c72d86c5b59a5106e408ec5a0ae30869c3b9688ca0a6"}
Sep 08 18:55:20.246840 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:20.246739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:31.253770 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:31.253736    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-865fc4c5d4-dwpl6"
Sep 08 18:55:31.273618 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:55:31.273561    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-865fc4c5d4-dwpl6" podStartSLOduration=11.929328264 podStartE2EDuration="12.273541978s" totalImagesPullingTime="344.213714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:55:20.264467216 +0000 UTC m=+1902.674777299" watchObservedRunningTime="2026-09-08 18:55:31.273541978 +0000 UTC m=+1913.683852059"
Sep 08 18:56:32.990052 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:32.989974    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"]
Sep 08 18:56:32.990865 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:32.990221    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd" podUID="4d13dee0-de08-472a-a28f-c8ffb2b51421" containerName="manager" containerID="cri-o://658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a" gracePeriod=10
Sep 08 18:56:33.354762 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.354732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:56:33.494333 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.494300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d13dee0-de08-472a-a28f-c8ffb2b51421-extensions-socket-volume\") pod \"4d13dee0-de08-472a-a28f-c8ffb2b51421\" (UID: \"4d13dee0-de08-472a-a28f-c8ffb2b51421\") "
Sep 08 18:56:33.494529 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.494396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jffjs\" (UniqueName: \"kubernetes.io/projected/4d13dee0-de08-472a-a28f-c8ffb2b51421-kube-api-access-jffjs\") pod \"4d13dee0-de08-472a-a28f-c8ffb2b51421\" (UID: \"4d13dee0-de08-472a-a28f-c8ffb2b51421\") "
Sep 08 18:56:33.494705 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.494681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d13dee0-de08-472a-a28f-c8ffb2b51421-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4d13dee0-de08-472a-a28f-c8ffb2b51421" (UID: "4d13dee0-de08-472a-a28f-c8ffb2b51421"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:56:33.496858 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.496833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d13dee0-de08-472a-a28f-c8ffb2b51421-kube-api-access-jffjs" (OuterVolumeSpecName: "kube-api-access-jffjs") pod "4d13dee0-de08-472a-a28f-c8ffb2b51421" (UID: "4d13dee0-de08-472a-a28f-c8ffb2b51421"). InnerVolumeSpecName "kube-api-access-jffjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:56:33.595525 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.595412    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d13dee0-de08-472a-a28f-c8ffb2b51421-extensions-socket-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:56:33.595525 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.595465    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jffjs\" (UniqueName: \"kubernetes.io/projected/4d13dee0-de08-472a-a28f-c8ffb2b51421-kube-api-access-jffjs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:56:33.597597 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.597569    2580 generic.go:356] "Generic (PLEG): container finished" podID="4d13dee0-de08-472a-a28f-c8ffb2b51421" containerID="658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a" exitCode=0
Sep 08 18:56:33.597714 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.597613    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"
Sep 08 18:56:33.597714 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.597612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd" event={"ID":"4d13dee0-de08-472a-a28f-c8ffb2b51421","Type":"ContainerDied","Data":"658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a"}
Sep 08 18:56:33.597794 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.597750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd" event={"ID":"4d13dee0-de08-472a-a28f-c8ffb2b51421","Type":"ContainerDied","Data":"cb189a69059a68b4c2c07a0d5b010ced8ea11805272c21a6996b6b1acfe8ac89"}
Sep 08 18:56:33.597794 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.597776    2580 scope.go:117] "RemoveContainer" containerID="658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a"
Sep 08 18:56:33.608385 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.608365    2580 scope.go:117] "RemoveContainer" containerID="658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a"
Sep 08 18:56:33.608684 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:56:33.608660    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a\": container with ID starting with 658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a not found: ID does not exist" containerID="658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a"
Sep 08 18:56:33.608757 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.608694    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a"} err="failed to get container status \"658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a\": rpc error: code = NotFound desc = could not find container \"658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a\": container with ID starting with 658a27adf60f76f2fa0cd62bfaf9203541f0cecce55a7fd657738f4da693294a not found: ID does not exist"
Sep 08 18:56:33.620279 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.620251    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"]
Sep 08 18:56:33.623672 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:33.623648    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-84vrd"]
Sep 08 18:56:34.120576 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:56:34.120546    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d13dee0-de08-472a-a28f-c8ffb2b51421" path="/var/lib/kubelet/pods/4d13dee0-de08-472a-a28f-c8ffb2b51421/volumes"
Sep 08 18:57:33.670350 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.670316    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"]
Sep 08 18:57:33.670824 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.670731    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d13dee0-de08-472a-a28f-c8ffb2b51421" containerName="manager"
Sep 08 18:57:33.670824 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.670743    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d13dee0-de08-472a-a28f-c8ffb2b51421" containerName="manager"
Sep 08 18:57:33.670824 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.670802    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d13dee0-de08-472a-a28f-c8ffb2b51421" containerName="manager"
Sep 08 18:57:33.674101 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.674084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:33.677356 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.677331    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9ld8f\""
Sep 08 18:57:33.688689 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.688655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"]
Sep 08 18:57:33.748833 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.748798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9frg\" (UniqueName: \"kubernetes.io/projected/815dae6a-00be-41ab-af85-d3264e30917e-kube-api-access-x9frg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pmw9d\" (UID: \"815dae6a-00be-41ab-af85-d3264e30917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:33.749026 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.748852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/815dae6a-00be-41ab-af85-d3264e30917e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pmw9d\" (UID: \"815dae6a-00be-41ab-af85-d3264e30917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:33.849482 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.849400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9frg\" (UniqueName: \"kubernetes.io/projected/815dae6a-00be-41ab-af85-d3264e30917e-kube-api-access-x9frg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pmw9d\" (UID: \"815dae6a-00be-41ab-af85-d3264e30917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:33.849482 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.849495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/815dae6a-00be-41ab-af85-d3264e30917e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pmw9d\" (UID: \"815dae6a-00be-41ab-af85-d3264e30917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:33.849955 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.849930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/815dae6a-00be-41ab-af85-d3264e30917e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pmw9d\" (UID: \"815dae6a-00be-41ab-af85-d3264e30917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:33.859139 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.859104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9frg\" (UniqueName: \"kubernetes.io/projected/815dae6a-00be-41ab-af85-d3264e30917e-kube-api-access-x9frg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pmw9d\" (UID: \"815dae6a-00be-41ab-af85-d3264e30917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:33.984634 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:33.984535    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:34.275340 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:34.275314    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"]
Sep 08 18:57:34.875987 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:34.875951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d" event={"ID":"815dae6a-00be-41ab-af85-d3264e30917e","Type":"ContainerStarted","Data":"25fa546733f2ac7a3b0ae8ac72116dc4da6a3f24e971405750756658f7a527b0"}
Sep 08 18:57:34.876416 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:34.876007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d" event={"ID":"815dae6a-00be-41ab-af85-d3264e30917e","Type":"ContainerStarted","Data":"4a3577f1a88ee38dcf75d3d47f88953ed31d491952e4328d465bb3ba6203bbe3"}
Sep 08 18:57:34.876416 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:34.876060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:57:34.900495 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:34.900408    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d" podStartSLOduration=1.900390447 podStartE2EDuration="1.900390447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:34.898019943 +0000 UTC m=+2037.308330033" watchObservedRunningTime="2026-09-08 18:57:34.900390447 +0000 UTC m=+2037.310700525"
Sep 08 18:57:45.884109 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:57:45.884077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pmw9d"
Sep 08 18:59:27.042265 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.042234    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"]
Sep 08 18:59:27.045655 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.045640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.055750 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.055729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"]
Sep 08 18:59:27.210663 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.210627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxfv\" (UniqueName: \"kubernetes.io/projected/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-kube-api-access-hrxfv\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.210849 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.210669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-openshift-service-ca\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.210849 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.210700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-metrics-certs\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.210849 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.210783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-maas-api-tls\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.311925 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.311825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-maas-api-tls\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.312082 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.311950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxfv\" (UniqueName: \"kubernetes.io/projected/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-kube-api-access-hrxfv\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.312082 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.311981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-openshift-service-ca\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.312082 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.312008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-metrics-certs\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.312808 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.312772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-openshift-service-ca\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.315172 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.315147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-maas-api-tls\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.315363 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.315339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-metrics-certs\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.320711 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.320688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxfv\" (UniqueName: \"kubernetes.io/projected/ef74a8a5-0cd4-4f7d-beb7-998278fc965d-kube-api-access-hrxfv\") pod \"maas-api-79cd597bc5-qlf8z\" (UID: \"ef74a8a5-0cd4-4f7d-beb7-998278fc965d\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.357286 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.357251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:27.631585 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:27.631549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"]
Sep 08 18:59:27.634671 ip-10-0-141-6 kubenswrapper[2580]: W0908 18:59:27.634644    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef74a8a5_0cd4_4f7d_beb7_998278fc965d.slice/crio-20ff904333f1cbf3f6eae3cca7aef1f6cface7f50a4c1b7ead9a056911071f67 WatchSource:0}: Error finding container 20ff904333f1cbf3f6eae3cca7aef1f6cface7f50a4c1b7ead9a056911071f67: Status 404 returned error can't find the container with id 20ff904333f1cbf3f6eae3cca7aef1f6cface7f50a4c1b7ead9a056911071f67
Sep 08 18:59:28.397967 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:28.397932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z" event={"ID":"ef74a8a5-0cd4-4f7d-beb7-998278fc965d","Type":"ContainerStarted","Data":"6cba49b30c6583edff83621992f90a178baae97ec16433a5324608423d689718"}
Sep 08 18:59:28.398387 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:28.397983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z" event={"ID":"ef74a8a5-0cd4-4f7d-beb7-998278fc965d","Type":"ContainerStarted","Data":"20ff904333f1cbf3f6eae3cca7aef1f6cface7f50a4c1b7ead9a056911071f67"}
Sep 08 18:59:28.398387 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:28.398010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:28.415395 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:28.415346    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z" podStartSLOduration=0.88414388 podStartE2EDuration="1.415329475s" totalImagesPullingTime="531.185595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:59:28.414664612 +0000 UTC m=+2150.824974696" watchObservedRunningTime="2026-09-08 18:59:28.415329475 +0000 UTC m=+2150.825639557"
Sep 08 18:59:34.409062 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:34.408977    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-qlf8z"
Sep 08 18:59:43.695542 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:43.695501    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"]
Sep 08 18:59:43.698083 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:43.695727    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s" podUID="1af19ccf-acde-4326-8146-513f4df9dcf9" containerName="maas-api" containerID="cri-o://2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3" gracePeriod=30
Sep 08 18:59:44.944581 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.944558    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:59:44.986038 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.986006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af19ccf-acde-4326-8146-513f4df9dcf9-openshift-service-ca\") pod \"1af19ccf-acde-4326-8146-513f4df9dcf9\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") "
Sep 08 18:59:44.986217 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.986112    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-maas-api-tls\") pod \"1af19ccf-acde-4326-8146-513f4df9dcf9\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") "
Sep 08 18:59:44.986217 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.986175    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-metrics-certs\") pod \"1af19ccf-acde-4326-8146-513f4df9dcf9\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") "
Sep 08 18:59:44.986217 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.986212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d7b7\" (UniqueName: \"kubernetes.io/projected/1af19ccf-acde-4326-8146-513f4df9dcf9-kube-api-access-7d7b7\") pod \"1af19ccf-acde-4326-8146-513f4df9dcf9\" (UID: \"1af19ccf-acde-4326-8146-513f4df9dcf9\") "
Sep 08 18:59:44.986460 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.986410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af19ccf-acde-4326-8146-513f4df9dcf9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1af19ccf-acde-4326-8146-513f4df9dcf9" (UID: "1af19ccf-acde-4326-8146-513f4df9dcf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:59:44.986608 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.986509    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af19ccf-acde-4326-8146-513f4df9dcf9-openshift-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:59:44.988478 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.988456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af19ccf-acde-4326-8146-513f4df9dcf9-kube-api-access-7d7b7" (OuterVolumeSpecName: "kube-api-access-7d7b7") pod "1af19ccf-acde-4326-8146-513f4df9dcf9" (UID: "1af19ccf-acde-4326-8146-513f4df9dcf9"). InnerVolumeSpecName "kube-api-access-7d7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:59:44.988570 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.988484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1af19ccf-acde-4326-8146-513f4df9dcf9" (UID: "1af19ccf-acde-4326-8146-513f4df9dcf9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:59:44.988570 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:44.988512    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1af19ccf-acde-4326-8146-513f4df9dcf9" (UID: "1af19ccf-acde-4326-8146-513f4df9dcf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:59:45.087890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.087802    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-maas-api-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:59:45.087890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.087833    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af19ccf-acde-4326-8146-513f4df9dcf9-metrics-certs\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:59:45.087890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.087844    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7b7\" (UniqueName: \"kubernetes.io/projected/1af19ccf-acde-4326-8146-513f4df9dcf9-kube-api-access-7d7b7\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\""
Sep 08 18:59:45.483808 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.483776    2580 generic.go:356] "Generic (PLEG): container finished" podID="1af19ccf-acde-4326-8146-513f4df9dcf9" containerID="2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3" exitCode=0
Sep 08 18:59:45.483990 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.483829    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"
Sep 08 18:59:45.483990 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.483843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s" event={"ID":"1af19ccf-acde-4326-8146-513f4df9dcf9","Type":"ContainerDied","Data":"2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3"}
Sep 08 18:59:45.483990 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.483890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s" event={"ID":"1af19ccf-acde-4326-8146-513f4df9dcf9","Type":"ContainerDied","Data":"925f33c7983e4e6a509eab4dfa69b94d1367c9b23e8710f063241a7d0dda459a"}
Sep 08 18:59:45.483990 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.483913    2580 scope.go:117] "RemoveContainer" containerID="2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3"
Sep 08 18:59:45.500162 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.500140    2580 scope.go:117] "RemoveContainer" containerID="2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3"
Sep 08 18:59:45.500475 ip-10-0-141-6 kubenswrapper[2580]: E0908 18:59:45.500454    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3\": container with ID starting with 2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3 not found: ID does not exist" containerID="2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3"
Sep 08 18:59:45.500563 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.500507    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3"} err="failed to get container status \"2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3\": rpc error: code = NotFound desc = could not find container \"2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3\": container with ID starting with 2f14ab943517d2b4a783b0c0226c012785c7070c11e2573d146ae8ba490cc5c3 not found: ID does not exist"
Sep 08 18:59:45.510890 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.510867    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"]
Sep 08 18:59:45.517019 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:45.516997    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-tqf5s"]
Sep 08 18:59:46.120759 ip-10-0-141-6 kubenswrapper[2580]: I0908 18:59:46.120721    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1af19ccf-acde-4326-8146-513f4df9dcf9" path="/var/lib/kubelet/pods/1af19ccf-acde-4326-8146-513f4df9dcf9/volumes"
Sep 08 19:00:02.503203 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:02.503158    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-865fc4c5d4-dwpl6_7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4/manager/0.log"
Sep 08 19:00:02.963387 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:02.963359    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8d688d88f-g5r25_eb6b9efa-19f3-42ea-8a09-560d57660a20/copy-manifests/0.log"
Sep 08 19:00:02.982662 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:02.982632    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8d688d88f-g5r25_eb6b9efa-19f3-42ea-8a09-560d57660a20/manager/0.log"
Sep 08 19:00:03.565182 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:03.565153    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79cd597bc5-qlf8z_ef74a8a5-0cd4-4f7d-beb7-998278fc965d/maas-api/0.log"
Sep 08 19:00:04.968756 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:04.968725    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-lxbp4_73cca14f-0dcc-4b7c-a2a5-acf5144bded6/manager/0.log"
Sep 08 19:00:05.212138 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:05.212108    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-pmw9d_815dae6a-00be-41ab-af85-d3264e30917e/manager/0.log"
Sep 08 19:00:05.922581 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:05.922546    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kw8r7_8e9facc1-6925-4fc1-b329-f8253a7a8c09/discovery/0.log"
Sep 08 19:00:07.003597 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:07.003566    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-slwwq_1b642420-8989-4339-a3cf-e427ee2894c6/main/0.log"
Sep 08 19:00:07.117421 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:07.117395    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zjqjt_0a95e427-7b33-4129-a07c-04d930a44236/main/0.log"
Sep 08 19:00:07.462482 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:07.462444    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fdbw_dd4a4e56-f6ab-4395-b3e3-31248364a19c/main/0.log"
Sep 08 19:00:16.797091 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:16.797058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m824h_c98280f8-b5ea-4ad5-b872-7c3d9f9379dd/global-pull-secret-syncer/0.log"
Sep 08 19:00:17.029950 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:17.029865    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xv525_cef15672-783f-42d2-a325-88f90a658e96/konnectivity-agent/0.log"
Sep 08 19:00:17.085664 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:17.085625    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-6.ec2.internal_19d2e81f86efd718848a5fc68affb80f/haproxy/0.log"
Sep 08 19:00:22.009867 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:22.009833    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-lxbp4_73cca14f-0dcc-4b7c-a2a5-acf5144bded6/manager/0.log"
Sep 08 19:00:22.133394 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:22.133315    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-pmw9d_815dae6a-00be-41ab-af85-d3264e30917e/manager/0.log"
Sep 08 19:00:23.811064 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:23.811032    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-f6b9v_40baa83d-6f8c-41f2-b74a-832246efe5cb/cluster-monitoring-operator/0.log"
Sep 08 19:00:23.852850 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:23.852824    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-5b6rv_42917c73-bc39-43cc-8538-d9301838cc0b/kube-state-metrics/0.log"
Sep 08 19:00:23.873138 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:23.873111    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-5b6rv_42917c73-bc39-43cc-8538-d9301838cc0b/kube-rbac-proxy-main/0.log"
Sep 08 19:00:23.896637 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:23.896616    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-5b6rv_42917c73-bc39-43cc-8538-d9301838cc0b/kube-rbac-proxy-self/0.log"
Sep 08 19:00:23.924329 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:23.924247    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54776484f4-hplz8_6860b446-b8b9-41cf-9e5b-12d7dc1f1514/metrics-server/0.log"
Sep 08 19:00:24.083795 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.083767    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lmh5d_c90e83c6-3962-4dd5-97fd-050165a9c846/node-exporter/0.log"
Sep 08 19:00:24.109046 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.109023    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lmh5d_c90e83c6-3962-4dd5-97fd-050165a9c846/kube-rbac-proxy/0.log"
Sep 08 19:00:24.132803 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.132774    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lmh5d_c90e83c6-3962-4dd5-97fd-050165a9c846/init-textfile/0.log"
Sep 08 19:00:24.377024 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.376999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90269e2-ebdb-417d-9071-da0467a09274/prometheus/0.log"
Sep 08 19:00:24.405581 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.405554    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90269e2-ebdb-417d-9071-da0467a09274/config-reloader/0.log"
Sep 08 19:00:24.462701 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.462672    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90269e2-ebdb-417d-9071-da0467a09274/thanos-sidecar/0.log"
Sep 08 19:00:24.486030 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.485999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90269e2-ebdb-417d-9071-da0467a09274/kube-rbac-proxy-web/0.log"
Sep 08 19:00:24.513494 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.513404    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90269e2-ebdb-417d-9071-da0467a09274/kube-rbac-proxy/0.log"
Sep 08 19:00:24.538523 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.538492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90269e2-ebdb-417d-9071-da0467a09274/kube-rbac-proxy-thanos/0.log"
Sep 08 19:00:24.560995 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.560966    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90269e2-ebdb-417d-9071-da0467a09274/init-config-reloader/0.log"
Sep 08 19:00:24.646109 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:24.646073    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-wbgc9_cf3e1995-8c3e-436c-83d3-ea2f38fcd01e/prometheus-operator-admission-webhook/0.log"
Sep 08 19:00:26.205538 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:26.205505    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-djllr_4fa09f9c-1d0e-427d-99fc-cc7c0aa637dc/networking-console-plugin/0.log"
Sep 08 19:00:27.021024 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.020983    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"]
Sep 08 19:00:27.021519 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.021491    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af19ccf-acde-4326-8146-513f4df9dcf9" containerName="maas-api"
Sep 08 19:00:27.021519 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.021517    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af19ccf-acde-4326-8146-513f4df9dcf9" containerName="maas-api"
Sep 08 19:00:27.021741 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.021621    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1af19ccf-acde-4326-8146-513f4df9dcf9" containerName="maas-api"
Sep 08 19:00:27.024902 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.024884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.027709 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.027690    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-phw82\"/\"kube-root-ca.crt\""
Sep 08 19:00:27.027709 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.027707    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-phw82\"/\"openshift-service-ca.crt\""
Sep 08 19:00:27.028682 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.028666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-phw82\"/\"default-dockercfg-fm6tw\""
Sep 08 19:00:27.034931 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.034911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"]
Sep 08 19:00:27.076943 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.076913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9mhg\" (UniqueName: \"kubernetes.io/projected/758cf730-2cd0-44c8-84cf-6b7107b745df-kube-api-access-m9mhg\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.077118 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.076949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-lib-modules\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.077118 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.076981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-sys\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.077118 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.077099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-podres\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.077247 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.077164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-proc\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178394 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-podres\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178625 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-proc\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178625 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9mhg\" (UniqueName: \"kubernetes.io/projected/758cf730-2cd0-44c8-84cf-6b7107b745df-kube-api-access-m9mhg\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178625 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-lib-modules\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178625 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-sys\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178625 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-proc\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178625 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-sys\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178625 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-podres\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.178905 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.178740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/758cf730-2cd0-44c8-84cf-6b7107b745df-lib-modules\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.187908 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.187884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9mhg\" (UniqueName: \"kubernetes.io/projected/758cf730-2cd0-44c8-84cf-6b7107b745df-kube-api-access-m9mhg\") pod \"perf-node-gather-daemonset-ld5lb\" (UID: \"758cf730-2cd0-44c8-84cf-6b7107b745df\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.336106 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.336002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:27.607211 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.607186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"]
Sep 08 19:00:27.609567 ip-10-0-141-6 kubenswrapper[2580]: W0908 19:00:27.609538    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod758cf730_2cd0_44c8_84cf_6b7107b745df.slice/crio-21c2f82861db098e2890d887f64e12be8cc0e893a345d451e043dcf3815c3156 WatchSource:0}: Error finding container 21c2f82861db098e2890d887f64e12be8cc0e893a345d451e043dcf3815c3156: Status 404 returned error can't find the container with id 21c2f82861db098e2890d887f64e12be8cc0e893a345d451e043dcf3815c3156
Sep 08 19:00:27.681723 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:27.681695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb" event={"ID":"758cf730-2cd0-44c8-84cf-6b7107b745df","Type":"ContainerStarted","Data":"21c2f82861db098e2890d887f64e12be8cc0e893a345d451e043dcf3815c3156"}
Sep 08 19:00:28.140311 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:28.140275    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-s75dr_07dc1c3a-0077-48b4-a8f7-c6700e878e85/volume-data-source-validator/0.log"
Sep 08 19:00:28.687665 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:28.687625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb" event={"ID":"758cf730-2cd0-44c8-84cf-6b7107b745df","Type":"ContainerStarted","Data":"e8d8195c2ca4056cf759dbbb03073c05356851f43ce6dce0a69bb53d7005b1cb"}
Sep 08 19:00:28.688080 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:28.687755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:28.704773 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:28.704724    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb" podStartSLOduration=1.704709039 podStartE2EDuration="1.704709039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 19:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:00:28.702543826 +0000 UTC m=+2211.112853908" watchObservedRunningTime="2026-09-08 19:00:28.704709039 +0000 UTC m=+2211.115019161"
Sep 08 19:00:29.240016 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:29.239990    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-44lg7_cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5/dns/0.log"
Sep 08 19:00:29.262049 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:29.262026    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-44lg7_cb8f8cfc-9c85-4e3d-ad27-7e45ff816ad5/kube-rbac-proxy/0.log"
Sep 08 19:00:29.389272 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:29.389239    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lgmvk_31debb12-8b51-43d8-8c51-4bba3cd84bcd/dns-node-resolver/0.log"
Sep 08 19:00:29.966975 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:29.966927    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cb99ddd65-v9zg2_7751a12e-8b39-4591-9bed-b91830663aa2/registry/0.log"
Sep 08 19:00:30.021195 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:30.021171    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gb8zd_3337f71d-84e5-4047-b6e3-614c0143b182/node-ca/0.log"
Sep 08 19:00:31.430048 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:31.430013    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kw8r7_8e9facc1-6925-4fc1-b329-f8253a7a8c09/discovery/0.log"
Sep 08 19:00:32.528258 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:32.528218    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7gf7v_97d26502-e9d6-4c71-92e3-46f65d798522/serve-healthcheck-canary/0.log"
Sep 08 19:00:33.253280 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:33.253248    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wnh7g_13e711ca-cb6f-4330-b5f8-4877403a29f9/kube-rbac-proxy/0.log"
Sep 08 19:00:33.275821 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:33.275789    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wnh7g_13e711ca-cb6f-4330-b5f8-4877403a29f9/exporter/0.log"
Sep 08 19:00:33.301044 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:33.301010    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wnh7g_13e711ca-cb6f-4330-b5f8-4877403a29f9/extractor/0.log"
Sep 08 19:00:34.703020 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:34.702992    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-ld5lb"
Sep 08 19:00:35.684835 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:35.684804    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-865fc4c5d4-dwpl6_7bd29fa6-0b1d-4e07-a0e0-746d0b5c4ae4/manager/0.log"
Sep 08 19:00:35.980238 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:35.980140    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8d688d88f-g5r25_eb6b9efa-19f3-42ea-8a09-560d57660a20/manager/0.log"
Sep 08 19:00:35.997414 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:35.997391    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8d688d88f-g5r25_eb6b9efa-19f3-42ea-8a09-560d57660a20/copy-manifests/0.log"
Sep 08 19:00:37.393512 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:37.393480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6b68db98f8-hltth_c084bd09-dd62-469d-8738-2ab54fe78e51/manager/0.log"
Sep 08 19:00:37.940760 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:37.940718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kw8r7_8e9facc1-6925-4fc1-b329-f8253a7a8c09/discovery/0.log"
Sep 08 19:00:39.995367 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:39.995335    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kw8r7_8e9facc1-6925-4fc1-b329-f8253a7a8c09/discovery/0.log"
Sep 08 19:00:42.992268 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:42.992231    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-mr28f_1bf50b32-0985-4f48-87a0-ef0b751950db/kube-storage-version-migrator-operator/1.log"
Sep 08 19:00:42.994040 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:42.994014    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-mr28f_1bf50b32-0985-4f48-87a0-ef0b751950db/kube-storage-version-migrator-operator/0.log"
Sep 08 19:00:44.074076 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.074048    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-29bnw_de1d37a6-9f03-44aa-be2f-ed48fbe1b40e/kube-multus/0.log"
Sep 08 19:00:44.339858 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.339781    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cvhnn_852cc4a9-da83-4816-a835-80bd44ada77f/kube-multus-additional-cni-plugins/0.log"
Sep 08 19:00:44.363276 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.363250    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cvhnn_852cc4a9-da83-4816-a835-80bd44ada77f/egress-router-binary-copy/0.log"
Sep 08 19:00:44.385721 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.385692    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cvhnn_852cc4a9-da83-4816-a835-80bd44ada77f/cni-plugins/0.log"
Sep 08 19:00:44.417559 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.417529    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cvhnn_852cc4a9-da83-4816-a835-80bd44ada77f/bond-cni-plugin/0.log"
Sep 08 19:00:44.445319 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.445256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cvhnn_852cc4a9-da83-4816-a835-80bd44ada77f/routeoverride-cni/0.log"
Sep 08 19:00:44.470878 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.470851    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cvhnn_852cc4a9-da83-4816-a835-80bd44ada77f/whereabouts-cni-bincopy/0.log"
Sep 08 19:00:44.493568 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.493541    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cvhnn_852cc4a9-da83-4816-a835-80bd44ada77f/whereabouts-cni/0.log"
Sep 08 19:00:44.821802 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.821758    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ndrsb_42d754f9-b01a-4e28-822e-79349dec9556/network-metrics-daemon/0.log"
Sep 08 19:00:44.839206 ip-10-0-141-6 kubenswrapper[2580]: I0908 19:00:44.839177    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ndrsb_42d754f9-b01a-4e28-822e-79349dec9556/kube-rbac-proxy/0.log"