Sep 07 11:20:47.353859 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 07 11:20:47.353875 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 07 11:20:47.353885 ip-10-0-143-241 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 07 11:20:47.354228 ip-10-0-143-241 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 07 11:20:57.603181 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 07 11:20:57.603198 ip-10-0-143-241 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d058854fdbaa40d1aff3dd93b4329fda --
Sep 07 11:22:30.185089 ip-10-0-143-241 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 11:22:30.613713 ip-10-0-143-241 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 11:22:30.613713 ip-10-0-143-241 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 11:22:30.613713 ip-10-0-143-241 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 11:22:30.613713 ip-10-0-143-241 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 11:22:30.613713 ip-10-0-143-241 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 11:22:30.616242 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.616160    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 11:22:30.622289 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622255    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:30.622289 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622275    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:30.622289 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622280    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:30.622289 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622290    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:30.622289 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622295    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:30.622289 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622300    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622304    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622308    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622311    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622315    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622319    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622323    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622327    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622331    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622335    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622338    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622342    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622346    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622349    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622353    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622357    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622361    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622367    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622373    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622378    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:30.622627 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622387    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622391    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622395    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622398    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622402    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622406    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622410    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622414    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622418    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622422    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622426    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622430    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622434    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622438    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622443    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622449    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622453    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622457    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622461    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622465    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:30.623459 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622469    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622474    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622477    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622482    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622486    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622494    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622500    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622505    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622510    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622515    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622519    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622523    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622527    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622532    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622536    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622541    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622545    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622549    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622554    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:30.624317 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622558    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622562    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622569    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622575    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622580    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622585    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622590    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622596    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622600    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622604    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622609    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622613    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622618    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622622    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622625    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622629    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622633    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622638    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622642    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:30.625126 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622647    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622652    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622656    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622660    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622666    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622671    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622675    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622680    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622684    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622688    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622692    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622697    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622702    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622706    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622710    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.622714    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623949    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623961    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623966    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623971    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:30.625963 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623975    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623980    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623985    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623990    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623994    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.623998    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624002    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624006    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624011    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624015    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624019    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624023    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624027    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624032    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624036    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624040    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624045    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624049    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624054    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624058    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:30.626894 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624061    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624066    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624070    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624075    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624079    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624083    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624087    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624091    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624095    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624100    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624104    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624107    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624113    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624117    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624122    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624128    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624135    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624140    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624147    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:30.627499 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624153    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624158    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624162    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624167    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624172    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624176    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624180    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624186    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624191    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624195    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624199    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624203    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624208    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624212    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624217    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624221    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624225    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624229    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624233    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624237    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:30.627998 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624242    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624246    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624250    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624254    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624258    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624263    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624267    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624271    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624275    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624279    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624285    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624290    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624294    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624298    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624302    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624306    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624310    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624314    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624318    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624321    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:30.628570 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624326    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624331    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624335    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624339    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624343    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624347    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624351    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624358    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624363    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624368    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624373    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624377    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624382    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624386    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624391    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624396    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.624993    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625002    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625009    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:30.629101 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625015    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625020    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625025    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625030    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625035    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625039    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625044    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625048    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625053    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625057    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625062    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625066    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625071    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625077    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625082    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625086    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625090    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625096    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625100    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:30.629646 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625105    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625109    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625113    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625117    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625122    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625126    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625130    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625133    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625137    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625141    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625145    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625150    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625154    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625158    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625164    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625170    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625174    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625178    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625182    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:30.630267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625186    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625190    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625194    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625199    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625203    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625210    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625216    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625221    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625226    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625231    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625235    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625240    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625244    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625248    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625253    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625258    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625262    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625267    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625271    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:30.630737 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625275    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625279    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625283    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625287    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625292    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625296    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625300    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625304    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625308    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625313    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625317    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625321    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625325    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625329    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625333    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625337    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625343    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625347    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625351    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625356    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:30.631234 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625360    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625364    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625369    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625373    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625377    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625381    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625385    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625389    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625393    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625397    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625401    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625405    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625409    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625413    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625420    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625425    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625429    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625433    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.625437    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625540    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625597    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 11:22:30.631720 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625606    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625613    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625619    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625624    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625631    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625637    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625642    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625647    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625652    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625658    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625665    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625670    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625675    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625679    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625685    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625690    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625697    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625702    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625707    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625712    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625717    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625723    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625728    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 11:22:30.632276 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625733    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625738    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625745    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625750    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625754    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625759    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625763    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625770    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625775    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625780    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625784    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625805    2581 flags.go:64] FLAG: --enable-server="true"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625810    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625817    2581 flags.go:64] FLAG: --event-burst="100"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625822    2581 flags.go:64] FLAG: --event-qps="50"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625826    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625831    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625835    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625841    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625848    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625852    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625857    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625862    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625866    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625872    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 11:22:30.632934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625877    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625881    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625886    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625891    2581 flags.go:64] FLAG: --feature-gates=""
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625897    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625902    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625907    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625912    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625916    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625923    2581 flags.go:64] FLAG: --help="false"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625928    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-241.ec2.internal"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625933    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625938    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625942    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625948    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625954    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625958    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625962    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625967    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625971    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625976    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625981    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625985    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625990    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 11:22:30.633572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.625994    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626000    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626006    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626011    2581 flags.go:64] FLAG: --lock-file=""
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626015    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626020    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626025    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626034    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626039    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626044    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626049    2581 flags.go:64] FLAG: --logging-format="text"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626053    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626058    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626063    2581 flags.go:64] FLAG: --manifest-url=""
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626068    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626074    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626079    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626085    2581 flags.go:64] FLAG: --max-pods="110"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626090    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626095    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626100    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626105    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626110    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626114    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626119    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 11:22:30.634182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626130    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626135    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626139    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626144    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626149    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626156    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626161    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626166    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626171    2581 flags.go:64] FLAG: --port="10250"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626175    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626181    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-041a75de992e74284"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626187    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626191    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626196    2581 flags.go:64] FLAG: --register-node="true"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626200    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626206    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626212    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626216    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626221    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626226    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626231    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626236    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626241    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626245    2581 flags.go:64] FLAG: --runonce="false"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626250    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 11:22:30.634818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626255    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626260    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626265    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626270    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626275    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626279    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626291    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626296    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626301    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626306    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626310    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626315    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626320    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626324    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626333    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626338    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626342    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626351    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626356    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626361    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626365    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626370    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626374    2581 flags.go:64] FLAG: --v="2"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626380    2581 flags.go:64] FLAG: --version="false"
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626387    2581 flags.go:64] FLAG: --vmodule=""
Sep 07 11:22:30.635440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626393    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.626399    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626602    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626610    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626615    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626620    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626625    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626630    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626634    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626638    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626643    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626647    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626651    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626655    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626659    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626663    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626667    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626671    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626675    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626680    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:30.636098 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626684    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626688    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626692    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626696    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626702    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626707    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626711    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626716    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626719    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626724    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626728    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626732    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626737    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626741    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626745    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626749    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626755    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626760    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626765    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:30.636587 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626769    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626773    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626777    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626783    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626808    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626814    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626819    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626824    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626829    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626833    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626838    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626842    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626847    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626851    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626855    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626859    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626864    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626870    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626874    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626878    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:30.637075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626882    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626886    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626890    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626895    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626898    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626902    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626907    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626911    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626915    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626919    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626923    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626930    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626934    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626938    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626942    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626946    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626950    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626955    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626959    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626963    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:30.637571 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626967    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626971    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626975    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626979    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626986    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626991    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.626996    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627001    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627006    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627013    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627017    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627022    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627026    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627031    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627036    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627041    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627045    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627049    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627053    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:30.638082 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627058    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627062    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.627067    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.628018    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.635566    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.635580    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.635600    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.635606    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635658    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635662    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635665    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635668    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635671    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635674    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635677    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:30.638566 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635680    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635683    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635686    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635689    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635691    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635694    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635696    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635699    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635701    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635704    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635706    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635709    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635711    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635714    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635716    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635719    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635722    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635724    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635726    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635729    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:30.639036 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635731    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635734    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635737    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635740    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635743    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635746    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635748    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635751    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635754    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635756    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635759    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635761    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635764    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635766    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635769    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635771    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635776    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635779    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635782    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:30.639555 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635785    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635803    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635806    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635809    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635813    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635816    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635819    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635821    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635824    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635827    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635830    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635832    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635835    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635839    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635842    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635846    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635850    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635853    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635856    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:30.640048 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635859    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635861    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635865    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635867    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635870    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635872    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635875    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635877    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635880    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635882    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635885    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635887    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635890    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635893    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635895    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635898    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635900    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635902    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635905    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635907    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:30.640507 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635910    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635912    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635915    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635917    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635920    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635922    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635925    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635927    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635931    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635934    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635938    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635942    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635945    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.635947    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.635952    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 11:22:30.641006 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636086    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636091    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636095    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636098    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636101    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636104    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636108    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636111    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636114    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636116    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636119    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636123    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636126    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636128    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636131    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636133    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636136    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636138    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636141    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636143    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 11:22:30.641393 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636146    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636148    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636151    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636154    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636156    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636159    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636162    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636164    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636168    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636170    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636172    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636175    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636177    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636180    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636182    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636185    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636187    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636189    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636192    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636194    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 11:22:30.641892 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636197    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636199    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636202    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636204    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636206    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636209    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636211    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636214    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636216    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636219    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636221    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636224    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636226    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636229    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636231    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636234    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636236    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636239    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636242    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636244    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 11:22:30.642408 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636247    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636250    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636252    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636255    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636257    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636259    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636262    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636264    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636267    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636269    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636272    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636274    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636278    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636281    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636285    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636287    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636290    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636293    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636296    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 11:22:30.643197 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636298    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636301    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636304    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636307    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636310    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636313    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636317    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636319    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636322    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636324    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636327    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636330    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636332    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636336    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636340    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636343    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636346    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636349    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636351    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 11:22:30.643648 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:30.636354    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 11:22:30.644143 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.636359    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 11:22:30.644143 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.636455    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 11:22:30.644143 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.638126    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 11:22:30.644143 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.639285    2581 server.go:1019] "Starting client certificate rotation"
Sep 07 11:22:30.644143 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.639380    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 11:22:30.644143 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.639413    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 11:22:30.644143 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.642174    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 11:22:30.648685 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.648663    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 11:22:30.657608 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.657589    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 07 11:22:30.662106 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.662089    2581 log.go:25] "Validated CRI v1 image API"
Sep 07 11:22:30.662969 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.662955    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 11:22:30.666311 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.666276    2581 fs.go:135] Filesystem UUIDs: map[1ca84f81-1255-4d3f-9d90-b6295b1130f8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bed5d79b-cb22-4e1e-9c84-10c2b7f8156e:/dev/nvme0n1p4]
Sep 07 11:22:30.666390 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.666310    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 11:22:30.671236 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.671129    2581 manager.go:216] Machine: {Timestamp:2026-09-07 11:22:30.669802344 +0000 UTC m=+0.375583024 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099305 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea4c4348e875b672c444344351270 SystemUUID:ec2ea4c4-348e-875b-672c-444344351270 BootID:d058854f-dbaa-40d1-aff3-dd93b4329fda Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e4:84:ae:6c:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e4:84:ae:6c:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:e3:37:24:c7:a9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 11:22:30.671236 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.671231    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 11:22:30.671338 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.671302    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 11:22:30.672163 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.672149    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 11:22:30.674772 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.674749    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 11:22:30.674926 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.674777    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 11:22:30.675654 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.675644    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 11:22:30.675689 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.675657    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 11:22:30.675689 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.675664    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 11:22:30.676594 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.676583    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 11:22:30.677851 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.677841    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 11:22:30.682642 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.682631    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 11:22:30.682744 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.682735    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 11:22:30.682828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.682764    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 11:22:30.682828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.682782    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 11:22:30.682828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.682815    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 11:22:30.682828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.682825    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 07 11:22:30.682967 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.682835    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 11:22:30.683132 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.683121    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 11:22:30.683167 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.683138    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 11:22:30.685719 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.685631    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 11:22:30.688842 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.688828    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 11:22:30.688889 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.688859    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 11:22:30.689668 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689655    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689676    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689686    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689692    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689701    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689707    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689722    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689728    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689735    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 11:22:30.689737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689741    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 11:22:30.690055 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689753    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 11:22:30.690055 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.689758    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 11:22:30.690625 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.690615    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 11:22:30.690625 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.690624    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 11:22:30.693906 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.693888    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 11:22:30.693990 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.693910    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 11:22:30.693990 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.693910    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 11:22:30.694591 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.694580    2581 server.go:1268] "Started kubelet"
Sep 07 11:22:30.694711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.694671    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 11:22:30.694770 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.694728    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 11:22:30.694843 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.694807    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 11:22:30.695261 ip-10-0-143-241 systemd[1]: Started Kubernetes Kubelet.
Sep 07 11:22:30.696146 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.696128    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 11:22:30.696842 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.696813    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 11:22:30.699818 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.699774    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 11:22:30.702582 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.702565    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 11:22:30.702668 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.702656    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 11:22:30.702733 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.702680    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:30.702864 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.701687    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-241.ec2.internal.18d30611d76b2d22  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-241.ec2.internal,UID:ip-10-0-143-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-241.ec2.internal,},FirstTimestamp:2026-09-07 11:22:30.694554914 +0000 UTC m=+0.400335596,LastTimestamp:2026-09-07 11:22:30.694554914 +0000 UTC m=+0.400335596,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-241.ec2.internal,}"
Sep 07 11:22:30.703301 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.703288    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 11:22:30.703301 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.703288    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 11:22:30.703424 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.703314    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 11:22:30.703424 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.703387    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 11:22:30.703424 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.703399    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 11:22:30.704656 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.704636    2581 factory.go:55] Registering systemd factory
Sep 07 11:22:30.704656 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.704654    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 07 11:22:30.705086 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.705068    2581 factory.go:153] Registering CRI-O factory
Sep 07 11:22:30.705170 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.705090    2581 factory.go:223] Registration of the crio container factory successfully
Sep 07 11:22:30.705170 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.705160    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 11:22:30.705269 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.705180    2581 factory.go:103] Registering Raw factory
Sep 07 11:22:30.705269 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.705197    2581 manager.go:1196] Started watching for new ooms in manager
Sep 07 11:22:30.705735 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.705711    2581 manager.go:349] Starting recovery of all containers
Sep 07 11:22:30.707713 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.707684    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 07 11:22:30.707843 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.707821    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 07 11:22:30.718392 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.718372    2581 manager.go:354] Recovery completed
Sep 07 11:22:30.721424 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.721401    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gts8x"
Sep 07 11:22:30.722500 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.722489    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:30.724752 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.724736    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:30.724838 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.724765    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:30.724838 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.724775    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:30.725309 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.725295    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 11:22:30.725309 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.725303    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:30.725309 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.725307    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 11:22:30.725469 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.725323    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 11:22:30.726070 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.726053    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:30.726130 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.726086    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:30.726130 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.726101    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:30.726756 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.726676    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-241.ec2.internal.18d30611d937f2b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-241.ec2.internal,UID:ip-10-0-143-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-241.ec2.internal,},FirstTimestamp:2026-09-07 11:22:30.724752049 +0000 UTC m=+0.430532730,LastTimestamp:2026-09-07 11:22:30.724752049 +0000 UTC m=+0.430532730,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-241.ec2.internal,}"
Sep 07 11:22:30.730253 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.730236    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gts8x"
Sep 07 11:22:30.734079 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.734019    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-241.ec2.internal.18d30611d938378f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-241.ec2.internal,UID:ip-10-0-143-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-241.ec2.internal,},FirstTimestamp:2026-09-07 11:22:30.724769679 +0000 UTC m=+0.430550360,LastTimestamp:2026-09-07 11:22:30.724769679 +0000 UTC m=+0.430550360,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-241.ec2.internal,}"
Sep 07 11:22:30.803114 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.803091    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:30.873910 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.873868    2581 policy_none.go:49] "None policy: Start"
Sep 07 11:22:30.873910 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.873895    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 11:22:30.873910 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.873909    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 11:22:30.876049 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.876036    2581 policy_none.go:47] "Start"
Sep 07 11:22:30.903256 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.903234    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:30.903330 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.903319    2581 manager.go:337] "Starting Device Plugin manager"
Sep 07 11:22:30.903367 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.903359    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 11:22:30.903408 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.903372    2581 server.go:87] "Starting device plugin registration server"
Sep 07 11:22:30.903622 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.903611    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 11:22:30.903671 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.903631    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 11:22:30.903778 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.903760    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 11:22:30.903893 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.903865    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 11:22:30.903893 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.903883    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 11:22:30.904466 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.904443    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 11:22:30.904541 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.904503    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:30.913337 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.913321    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal"
Sep 07 11:22:30.933065 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.932931    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 11:22:30.934259 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.934239    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 11:22:30.934344 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.934293    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 11:22:30.934382 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.934362    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 11:22:30.934462 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:30.934446    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 11:22:30.936885 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:30.936871    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:31.004190 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.004161    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:31.005130 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.005115    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:31.005206 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.005149    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:31.005206 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.005178    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:31.005282 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.005226    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.014710 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.014694    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.014772 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.014713    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-241.ec2.internal\": node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.035158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.035131    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"]
Sep 07 11:22:31.035257 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.035196    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:31.036529 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.036513    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:31.036633 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.036541    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:31.036633 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.036562    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:31.038967 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.038953    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:31.039072 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.039059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.039156 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.039098    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:31.039637 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.039611    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:31.039691 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.039642    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:31.039691 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.039658    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:31.039691 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.039663    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:31.039691 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.039678    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:31.039691 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.039689    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:31.041959 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.041940    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.042051 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.041971    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:31.043295 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.043281    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:31.043363 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.043305    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:31.043363 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.043315    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:31.065391 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.065365    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.065640 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.065624    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.070160 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.070145    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.104304 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.104279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.104304 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.104304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20ada389393df0af1b24f49af160350a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"20ada389393df0af1b24f49af160350a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.104432 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.104319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.165763 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.165705    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.205229 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.205206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.205303 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.205232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.205303 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.205261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20ada389393df0af1b24f49af160350a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"20ada389393df0af1b24f49af160350a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.205373 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.205315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.205373 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.205342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20ada389393df0af1b24f49af160350a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"20ada389393df0af1b24f49af160350a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.205373 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.205315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.266361 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.266325    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.366803 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.366764    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.366885 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.366807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.373559 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.373543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.466888 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.466850    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.568785 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.568743    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.639369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.639336    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 11:22:31.640085 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.639506    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 11:22:31.670901 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.670874    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.736499 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.736403    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 11:17:31 +0000 UTC" deadline="2028-06-15 02:51:55.66598304 +0000 UTC"
Sep 07 11:22:31.736499 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.736442    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15519h29m23.929545986s"
Sep 07 11:22:31.771587 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:31.771544    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 07 11:22:31.831563 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.831539    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 11:22:31.831704 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.831569    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:31.832466 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.832450    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 11:22:31.832542 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.832479    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 11:22:31.832542 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.832490    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 07 11:22:31.887750 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.887726    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:31.903587 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.903562    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.916443 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.916422    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 11:22:31.918128 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.918114    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 07 11:22:31.927140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:31.927124    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 11:22:32.127079 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:32.127039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ada389393df0af1b24f49af160350a.slice/crio-3520bafd60078c2bddac966abe7278d654f72bbb641d000ab38d83ae0f11d2d8 WatchSource:0}: Error finding container 3520bafd60078c2bddac966abe7278d654f72bbb641d000ab38d83ae0f11d2d8: Status 404 returned error can't find the container with id 3520bafd60078c2bddac966abe7278d654f72bbb641d000ab38d83ae0f11d2d8
Sep 07 11:22:32.127276 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:32.127255    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6999f592201b6868f277532693fdb1.slice/crio-dee7e34151d13eafd5c07c8daee3b0a2aacab68ec4c85ca9cfc0a5c768eb41e8 WatchSource:0}: Error finding container dee7e34151d13eafd5c07c8daee3b0a2aacab68ec4c85ca9cfc0a5c768eb41e8: Status 404 returned error can't find the container with id dee7e34151d13eafd5c07c8daee3b0a2aacab68ec4c85ca9cfc0a5c768eb41e8
Sep 07 11:22:32.132363 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.132343    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:22:32.191309 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.191285    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:32.684590 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.684556    2581 apiserver.go:52] "Watching apiserver"
Sep 07 11:22:32.691655 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.691636    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 11:22:32.692065 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.692045    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-sh9c5","openshift-ovn-kubernetes/ovnkube-node-prn2z","kube-system/global-pull-secret-syncer-nkt94","kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7w9gb","openshift-dns/node-resolver-m8j27","openshift-image-registry/node-ca-lt54v","openshift-multus/network-metrics-daemon-lmtxp","openshift-network-diagnostics/network-check-target-jk5dm","kube-system/konnectivity-agent-z4xbr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal","openshift-multus/multus-additional-cni-plugins-9hpnc","openshift-multus/multus-xpwmp"]
Sep 07 11:22:32.696150 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.696130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.698643 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.698613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.698979 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.698818    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:22:32.699103 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.698981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h8gjf\""
Sep 07 11:22:32.699103 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.699022    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 11:22:32.701197 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.701174    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 11:22:32.701310 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.701283    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 11:22:32.701465 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.701448    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 11:22:32.702151 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.701700    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8kv7f\""
Sep 07 11:22:32.702151 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.701948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 11:22:32.702151 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.701964    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 11:22:32.702369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.702183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.702369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.702223    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 11:22:32.703053 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.703028    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 11:22:32.704341 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.704117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:32.704341 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.704203    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:32.704630 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.704613    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 11:22:32.704708 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.704590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gtd26\""
Sep 07 11:22:32.704932 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.704909    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 11:22:32.704932 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.704923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 11:22:32.707061 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.707043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.709191 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.709170    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 11:22:32.709314 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.709298    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 11:22:32.709467 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.709449    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 11:22:32.709549 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.709531    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z9frs\""
Sep 07 11:22:32.709663 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.709649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.712071 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.712050    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.712375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.712351    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wqspj\""
Sep 07 11:22:32.712455 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.712379    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 11:22:32.712631 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.712615    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 11:22:32.714349 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-sys\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.714450 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-cni-bin\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714450 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d88b\" (UniqueName: \"kubernetes.io/projected/ab8aa650-17bb-4302-8c52-177600aae204-kube-api-access-4d88b\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714450 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-socket-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.714450 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-var-lib-kubelet\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.714649 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-kubernetes\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.714649 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-lib-modules\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.714649 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-kubelet\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714649 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-ovn\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714649 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714649 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.714688    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-log-socket\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0427ce89-4926-4887-a9fe-6d49fb517836-etc-tuned\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0427ce89-4926-4887-a9fe-6d49fb517836-tmp\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-var-lib-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-etc-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.714974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab8aa650-17bb-4302-8c52-177600aae204-ovn-node-metrics-cert\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.714958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9dc\" (UniqueName: \"kubernetes.io/projected/0db155b2-82b9-471a-b334-4c7b1ee71288-kube-api-access-wk9dc\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysctl-d\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-systemd\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-device-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69nk4\" (UniqueName: \"kubernetes.io/projected/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-kube-api-access-69nk4\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715128    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j27fp\""
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bss\" (UniqueName: \"kubernetes.io/projected/0427ce89-4926-4887-a9fe-6d49fb517836-kube-api-access-p5bss\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-run-netns\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-cni-netd\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-systemd-units\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-run-ovn-kubernetes\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-env-overrides\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-ovnkube-script-lib\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.715340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-registration-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysctl-conf\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-run\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-host\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-slash\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-ovnkube-config\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-sys-fs\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-node-log\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-modprobe-d\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysconfig\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-systemd\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.716057 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.715726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:22:32.716494 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.716474    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 11:22:32.716568 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.716551    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 11:22:32.717099 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.717083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:32.719333 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.719311    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 11:22:32.719422 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.719410    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 11:22:32.719670 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.719636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.719767 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.719675    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t2m96\""
Sep 07 11:22:32.719767 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.719697    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:32.719767 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.719706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 11:22:32.722304 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.722287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.724765 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.724741    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 11:22:32.724879 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.724849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.724949 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.724748    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bv9nt\""
Sep 07 11:22:32.724949 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.724939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 11:22:32.725344 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.725323    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 11:22:32.725427 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.725353    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 11:22:32.725427 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.725325    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 11:22:32.728239 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.728211    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 11:22:32.728395 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.728375    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rdtd4\""
Sep 07 11:22:32.738138 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.738122    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-htvdx"
Sep 07 11:22:32.748915 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.748897    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-htvdx"
Sep 07 11:22:32.804705 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.804680    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 11:22:32.816099 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysconfig\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.816172 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-systemd\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816172 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fbf\" (UniqueName: \"kubernetes.io/projected/b1a3c173-d443-479b-9521-edf01ab367e8-kube-api-access-d7fbf\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.816172 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-etc-kubernetes\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.816172 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-serviceca\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.816375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-tmp-dir\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.816375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysconfig\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.816375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-systemd\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-sys\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.816375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-cni-bin\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-cni-bin\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-sys\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.816556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-kubelet\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-ovn\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-kubelet\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-ovn\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.816556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9dc\" (UniqueName: \"kubernetes.io/projected/0db155b2-82b9-471a-b334-4c7b1ee71288-kube-api-access-wk9dc\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-cni-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-socket-dir-parent\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0427ce89-4926-4887-a9fe-6d49fb517836-etc-tuned\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0427ce89-4926-4887-a9fe-6d49fb517836-tmp\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-var-lib-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-conf-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysctl-d\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-var-lib-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nk4\" (UniqueName: \"kubernetes.io/projected/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-kube-api-access-69nk4\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-run-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40b2c47e-d219-4889-b4ed-d78eaa2f179e-konnectivity-ca\") pod \"konnectivity-agent-z4xbr\" (UID: \"40b2c47e-d219-4889-b4ed-d78eaa2f179e\") " pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wwjb\" (UniqueName: \"kubernetes.io/projected/43925274-591e-4a1c-972f-aba80b98abbf-kube-api-access-8wwjb\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.816986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-run-netns\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysctl-d\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-cni-netd\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.816985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-run-netns\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817018    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-cni-netd\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-systemd-units\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-ovnkube-script-lib\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817094    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-registration-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-systemd-units\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43925274-591e-4a1c-972f-aba80b98abbf-multus-daemon-config\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-registration-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-multus-certs\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-dbus\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysctl-conf\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-ovnkube-config\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.817817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-node-log\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-sysctl-conf\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-host\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ckmn\" (UniqueName: \"kubernetes.io/projected/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-kube-api-access-4ckmn\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-node-log\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-kubelet-config\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d88b\" (UniqueName: \"kubernetes.io/projected/ab8aa650-17bb-4302-8c52-177600aae204-kube-api-access-4d88b\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-socket-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-cnibin\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3909660c-5e60-493f-b6fa-79ad570b4d90-host-slash\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40b2c47e-d219-4889-b4ed-d78eaa2f179e-agent-certs\") pod \"konnectivity-agent-z4xbr\" (UID: \"40b2c47e-d219-4889-b4ed-d78eaa2f179e\") " pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-system-cni-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cg29\" (UniqueName: \"kubernetes.io/projected/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-kube-api-access-8cg29\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-ovnkube-script-lib\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-var-lib-kubelet\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-kubernetes\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.818646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-lib-modules\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-var-lib-kubelet\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-socket-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43925274-591e-4a1c-972f-aba80b98abbf-cni-binary-copy\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-ovnkube-config\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-kubernetes\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-cni-bin\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-hostroot\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-lib-modules\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.817979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-log-socket\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-etc-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab8aa650-17bb-4302-8c52-177600aae204-ovn-node-metrics-cert\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-log-socket\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.819339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-etc-openvswitch\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-systemd\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-device-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-systemd\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-cnibin\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bss\" (UniqueName: \"kubernetes.io/projected/0427ce89-4926-4887-a9fe-6d49fb517836-kube-api-access-p5bss\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-device-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.818283    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2t29\" (UniqueName: \"kubernetes.io/projected/3909660c-5e60-493f-b6fa-79ad570b4d90-kube-api-access-g2t29\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-k8s-cni-cncf-io\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.818374    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs podName:85e5bbff-0cc8-45cb-91b8-b106b0a5a978 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:33.318353717 +0000 UTC m=+3.024134388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs") pod "network-metrics-daemon-lmtxp" (UID: "85e5bbff-0cc8-45cb-91b8-b106b0a5a978") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-netns\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-kubelet\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-run-ovn-kubernetes\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.819917 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-env-overrides\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-run-ovn-kubernetes\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-cni-multus\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-hosts-file\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-run\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-host\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-slash\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-sys-fs\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3909660c-5e60-493f-b6fa-79ad570b4d90-iptables-alerter-script\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-host\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-modprobe-d\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-run\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0db155b2-82b9-471a-b334-4c7b1ee71288-sys-fs\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0427ce89-4926-4887-a9fe-6d49fb517836-etc-modprobe-d\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab8aa650-17bb-4302-8c52-177600aae204-env-overrides\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.818903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab8aa650-17bb-4302-8c52-177600aae204-host-slash\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.820640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.820564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0427ce89-4926-4887-a9fe-6d49fb517836-tmp\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.821150 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.820783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0427ce89-4926-4887-a9fe-6d49fb517836-etc-tuned\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.821150 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.820943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab8aa650-17bb-4302-8c52-177600aae204-ovn-node-metrics-cert\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.826464 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.826443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9dc\" (UniqueName: \"kubernetes.io/projected/0db155b2-82b9-471a-b334-4c7b1ee71288-kube-api-access-wk9dc\") pod \"aws-ebs-csi-driver-node-gmzsl\" (UID: \"0db155b2-82b9-471a-b334-4c7b1ee71288\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:32.826598 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.826581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nk4\" (UniqueName: \"kubernetes.io/projected/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-kube-api-access-69nk4\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:32.828554 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.828536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d88b\" (UniqueName: \"kubernetes.io/projected/ab8aa650-17bb-4302-8c52-177600aae204-kube-api-access-4d88b\") pod \"ovnkube-node-prn2z\" (UID: \"ab8aa650-17bb-4302-8c52-177600aae204\") " pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:32.828748 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.828729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bss\" (UniqueName: \"kubernetes.io/projected/0427ce89-4926-4887-a9fe-6d49fb517836-kube-api-access-p5bss\") pod \"tuned-7w9gb\" (UID: \"0427ce89-4926-4887-a9fe-6d49fb517836\") " pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:32.883808 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.883765    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 11:22:32.919356 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3909660c-5e60-493f-b6fa-79ad570b4d90-iptables-alerter-script\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.919498 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fbf\" (UniqueName: \"kubernetes.io/projected/b1a3c173-d443-479b-9521-edf01ab367e8-kube-api-access-d7fbf\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.919498 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-etc-kubernetes\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.919498 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-serviceca\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.919703 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-etc-kubernetes\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.919830 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-tmp-dir\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.919830 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-cni-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.919830 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-socket-dir-parent\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.920015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-conf-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.920015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:32.920015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40b2c47e-d219-4889-b4ed-d78eaa2f179e-konnectivity-ca\") pod \"konnectivity-agent-z4xbr\" (UID: \"40b2c47e-d219-4889-b4ed-d78eaa2f179e\") " pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:32.920015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wwjb\" (UniqueName: \"kubernetes.io/projected/43925274-591e-4a1c-972f-aba80b98abbf-kube-api-access-8wwjb\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.920015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-serviceca\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.920015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.919995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-conf-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43925274-591e-4a1c-972f-aba80b98abbf-multus-daemon-config\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-cni-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-multus-certs\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-dbus\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-host\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ckmn\" (UniqueName: \"kubernetes.io/projected/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-kube-api-access-4ckmn\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-kubelet-config\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-cnibin\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-multus-socket-dir-parent\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3909660c-5e60-493f-b6fa-79ad570b4d90-iptables-alerter-script\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3909660c-5e60-493f-b6fa-79ad570b4d90-host-slash\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.920383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40b2c47e-d219-4889-b4ed-d78eaa2f179e-agent-certs\") pod \"konnectivity-agent-z4xbr\" (UID: \"40b2c47e-d219-4889-b4ed-d78eaa2f179e\") " pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-tmp-dir\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-system-cni-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-kubelet-config\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cg29\" (UniqueName: \"kubernetes.io/projected/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-kube-api-access-8cg29\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-host\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3909660c-5e60-493f-b6fa-79ad570b4d90-host-slash\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43925274-591e-4a1c-972f-aba80b98abbf-cni-binary-copy\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-cni-bin\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-system-cni-dir\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-hostroot\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43925274-591e-4a1c-972f-aba80b98abbf-multus-daemon-config\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a3c173-d443-479b-9521-edf01ab367e8-cnibin\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-cnibin\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-multus-certs\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-cnibin\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-hostroot\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-dbus\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2t29\" (UniqueName: \"kubernetes.io/projected/3909660c-5e60-493f-b6fa-79ad570b4d90-kube-api-access-g2t29\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.920995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-k8s-cni-cncf-io\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-netns\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-kubelet\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-cni-multus\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-hosts-file\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-hosts-file\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43925274-591e-4a1c-972f-aba80b98abbf-cni-binary-copy\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-k8s-cni-cncf-io\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.921761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-run-netns\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.922423 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.921289    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:32.922423 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-cni-multus\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.922423 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.921393    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret podName:3d0f4eb6-2bba-480e-8dae-1827c5ac731a nodeName:}" failed. No retries permitted until 2026-09-07 11:22:33.421375448 +0000 UTC m=+3.127156145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret") pod "global-pull-secret-syncer-nkt94" (UID: "3d0f4eb6-2bba-480e-8dae-1827c5ac731a") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:32.922423 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-kubelet\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.922423 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43925274-591e-4a1c-972f-aba80b98abbf-host-var-lib-cni-bin\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.922423 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a3c173-d443-479b-9521-edf01ab367e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.922423 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.921785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40b2c47e-d219-4889-b4ed-d78eaa2f179e-konnectivity-ca\") pod \"konnectivity-agent-z4xbr\" (UID: \"40b2c47e-d219-4889-b4ed-d78eaa2f179e\") " pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:32.923334 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.923305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40b2c47e-d219-4889-b4ed-d78eaa2f179e-agent-certs\") pod \"konnectivity-agent-z4xbr\" (UID: \"40b2c47e-d219-4889-b4ed-d78eaa2f179e\") " pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:32.930172 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.930123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fbf\" (UniqueName: \"kubernetes.io/projected/b1a3c173-d443-479b-9521-edf01ab367e8-kube-api-access-d7fbf\") pod \"multus-additional-cni-plugins-9hpnc\" (UID: \"b1a3c173-d443-479b-9521-edf01ab367e8\") " pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:32.931914 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.931891    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:32.932023 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.931920    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:32.932023 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.931934    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ntf9g for pod openshift-network-diagnostics/network-check-target-jk5dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:32.932123 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:32.932041    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g podName:caa455b9-a713-4cbb-8d32-26da115b04d1 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:33.432024229 +0000 UTC m=+3.137804918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ntf9g" (UniqueName: "kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g") pod "network-check-target-jk5dm" (UID: "caa455b9-a713-4cbb-8d32-26da115b04d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:32.934516 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.934493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cg29\" (UniqueName: \"kubernetes.io/projected/52a43c8b-4eaf-4514-a60f-d0fba1696bd0-kube-api-access-8cg29\") pod \"node-ca-lt54v\" (UID: \"52a43c8b-4eaf-4514-a60f-d0fba1696bd0\") " pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:32.934817 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.934763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2t29\" (UniqueName: \"kubernetes.io/projected/3909660c-5e60-493f-b6fa-79ad570b4d90-kube-api-access-g2t29\") pod \"iptables-alerter-sh9c5\" (UID: \"3909660c-5e60-493f-b6fa-79ad570b4d90\") " pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:32.937039 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.936897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wwjb\" (UniqueName: \"kubernetes.io/projected/43925274-591e-4a1c-972f-aba80b98abbf-kube-api-access-8wwjb\") pod \"multus-xpwmp\" (UID: \"43925274-591e-4a1c-972f-aba80b98abbf\") " pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:32.937039 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.936930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ckmn\" (UniqueName: \"kubernetes.io/projected/dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1-kube-api-access-4ckmn\") pod \"node-resolver-m8j27\" (UID: \"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1\") " pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:32.941147 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.940921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"9a6999f592201b6868f277532693fdb1","Type":"ContainerStarted","Data":"dee7e34151d13eafd5c07c8daee3b0a2aacab68ec4c85ca9cfc0a5c768eb41e8"}
Sep 07 11:22:32.942518 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:32.942492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" event={"ID":"20ada389393df0af1b24f49af160350a","Type":"ContainerStarted","Data":"3520bafd60078c2bddac966abe7278d654f72bbb641d000ab38d83ae0f11d2d8"}
Sep 07 11:22:33.009118 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.009089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7w9gb"
Sep 07 11:22:33.015326 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.015301    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:33.023781 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.023756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl"
Sep 07 11:22:33.028219 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.028197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lt54v"
Sep 07 11:22:33.035601 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.034750    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m8j27"
Sep 07 11:22:33.038111 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:33.038077    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db155b2_82b9_471a_b334_4c7b1ee71288.slice/crio-74ed0ce161a143a3efa8ad9a4074a2092fbe1c9d086b9c3699d920cc9dfdaa79 WatchSource:0}: Error finding container 74ed0ce161a143a3efa8ad9a4074a2092fbe1c9d086b9c3699d920cc9dfdaa79: Status 404 returned error can't find the container with id 74ed0ce161a143a3efa8ad9a4074a2092fbe1c9d086b9c3699d920cc9dfdaa79
Sep 07 11:22:33.040565 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.040544    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sh9c5"
Sep 07 11:22:33.040788 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:33.040756    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a43c8b_4eaf_4514_a60f_d0fba1696bd0.slice/crio-6228f02f02fd46dd5a5d60d0eac86582fbe516fda5f8d19c67240ebc4554ec47 WatchSource:0}: Error finding container 6228f02f02fd46dd5a5d60d0eac86582fbe516fda5f8d19c67240ebc4554ec47: Status 404 returned error can't find the container with id 6228f02f02fd46dd5a5d60d0eac86582fbe516fda5f8d19c67240ebc4554ec47
Sep 07 11:22:33.046842 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.046820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:33.048168 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:33.047661    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce8b7bd_5f71_4d33_a6f5_dfe8ae3c02b1.slice/crio-23cc5cea71bf0bda9f5793b20e5d5774c841185bd3a534f834b673f61b1f8848 WatchSource:0}: Error finding container 23cc5cea71bf0bda9f5793b20e5d5774c841185bd3a534f834b673f61b1f8848: Status 404 returned error can't find the container with id 23cc5cea71bf0bda9f5793b20e5d5774c841185bd3a534f834b673f61b1f8848
Sep 07 11:22:33.052372 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.052345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9hpnc"
Sep 07 11:22:33.052634 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:33.052603    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3909660c_5e60_493f_b6fa_79ad570b4d90.slice/crio-325b71a9af126f05904358e8f519d54b10d8d262300bd6e1572ecb528e94ff7d WatchSource:0}: Error finding container 325b71a9af126f05904358e8f519d54b10d8d262300bd6e1572ecb528e94ff7d: Status 404 returned error can't find the container with id 325b71a9af126f05904358e8f519d54b10d8d262300bd6e1572ecb528e94ff7d
Sep 07 11:22:33.058615 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.058461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xpwmp"
Sep 07 11:22:33.063275 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:33.063223    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a3c173_d443_479b_9521_edf01ab367e8.slice/crio-feb9324656f8765403f3c8686814d32e6fae050201e4697beeb08b02c3f5b8a3 WatchSource:0}: Error finding container feb9324656f8765403f3c8686814d32e6fae050201e4697beeb08b02c3f5b8a3: Status 404 returned error can't find the container with id feb9324656f8765403f3c8686814d32e6fae050201e4697beeb08b02c3f5b8a3
Sep 07 11:22:33.063619 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:22:33.063598    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b2c47e_d219_4889_b4ed_d78eaa2f179e.slice/crio-4c72127b93328283abfdfba87cca90aabef085af18b3e81d0b95a380028b568a WatchSource:0}: Error finding container 4c72127b93328283abfdfba87cca90aabef085af18b3e81d0b95a380028b568a: Status 404 returned error can't find the container with id 4c72127b93328283abfdfba87cca90aabef085af18b3e81d0b95a380028b568a
Sep 07 11:22:33.325636 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.325596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:33.325816 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:33.325785    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:33.325946 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:33.325892    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs podName:85e5bbff-0cc8-45cb-91b8-b106b0a5a978 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:34.325871966 +0000 UTC m=+4.031652635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs") pod "network-metrics-daemon-lmtxp" (UID: "85e5bbff-0cc8-45cb-91b8-b106b0a5a978") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:33.426716 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.426615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:33.426927 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:33.426783    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:33.426927 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:33.426881    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret podName:3d0f4eb6-2bba-480e-8dae-1827c5ac731a nodeName:}" failed. No retries permitted until 2026-09-07 11:22:34.426859071 +0000 UTC m=+4.132639752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret") pod "global-pull-secret-syncer-nkt94" (UID: "3d0f4eb6-2bba-480e-8dae-1827c5ac731a") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:33.527126 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.527085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:33.527305 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:33.527253    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:33.527305 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:33.527272    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:33.527305 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:33.527286    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ntf9g for pod openshift-network-diagnostics/network-check-target-jk5dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:33.527431 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:33.527359    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g podName:caa455b9-a713-4cbb-8d32-26da115b04d1 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:34.527338882 +0000 UTC m=+4.233119568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntf9g" (UniqueName: "kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g") pod "network-check-target-jk5dm" (UID: "caa455b9-a713-4cbb-8d32-26da115b04d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:33.752628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.752561    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 11:17:33 +0000 UTC" deadline="2028-04-12 03:36:59.610571523 +0000 UTC"
Sep 07 11:22:33.752628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.752623    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13984h14m25.85795193s"
Sep 07 11:22:33.946353 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.946316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xpwmp" event={"ID":"43925274-591e-4a1c-972f-aba80b98abbf","Type":"ContainerStarted","Data":"ff4e84562059997e32853ed4565c26bc412482e779c00ed311cbc0fdb21eae2c"}
Sep 07 11:22:33.947801 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.947771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" event={"ID":"b1a3c173-d443-479b-9521-edf01ab367e8","Type":"ContainerStarted","Data":"feb9324656f8765403f3c8686814d32e6fae050201e4697beeb08b02c3f5b8a3"}
Sep 07 11:22:33.949261 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.949231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z4xbr" event={"ID":"40b2c47e-d219-4889-b4ed-d78eaa2f179e","Type":"ContainerStarted","Data":"4c72127b93328283abfdfba87cca90aabef085af18b3e81d0b95a380028b568a"}
Sep 07 11:22:33.950744 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.950714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lt54v" event={"ID":"52a43c8b-4eaf-4514-a60f-d0fba1696bd0","Type":"ContainerStarted","Data":"6228f02f02fd46dd5a5d60d0eac86582fbe516fda5f8d19c67240ebc4554ec47"}
Sep 07 11:22:33.952077 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.952049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl" event={"ID":"0db155b2-82b9-471a-b334-4c7b1ee71288","Type":"ContainerStarted","Data":"74ed0ce161a143a3efa8ad9a4074a2092fbe1c9d086b9c3699d920cc9dfdaa79"}
Sep 07 11:22:33.953452 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.953421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7w9gb" event={"ID":"0427ce89-4926-4887-a9fe-6d49fb517836","Type":"ContainerStarted","Data":"d0378330d037e26b7e5ed4432ecdf537469e7fa41b778c0be494ad8a3e85c477"}
Sep 07 11:22:33.954837 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.954806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sh9c5" event={"ID":"3909660c-5e60-493f-b6fa-79ad570b4d90","Type":"ContainerStarted","Data":"325b71a9af126f05904358e8f519d54b10d8d262300bd6e1572ecb528e94ff7d"}
Sep 07 11:22:33.956105 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.956081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m8j27" event={"ID":"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1","Type":"ContainerStarted","Data":"23cc5cea71bf0bda9f5793b20e5d5774c841185bd3a534f834b673f61b1f8848"}
Sep 07 11:22:33.957478 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:33.957455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"d4cc7df9cf1a4cbf931a0b91920165926def1b81e45c75be0fefa1a06c523166"}
Sep 07 11:22:34.334224 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:34.333628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:34.334224 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.333771    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:34.334224 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.333856    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs podName:85e5bbff-0cc8-45cb-91b8-b106b0a5a978 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:36.333816252 +0000 UTC m=+6.039596928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs") pod "network-metrics-daemon-lmtxp" (UID: "85e5bbff-0cc8-45cb-91b8-b106b0a5a978") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:34.434668 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:34.434626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:34.434902 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.434807    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:34.434981 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.434923    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret podName:3d0f4eb6-2bba-480e-8dae-1827c5ac731a nodeName:}" failed. No retries permitted until 2026-09-07 11:22:36.434901527 +0000 UTC m=+6.140682212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret") pod "global-pull-secret-syncer-nkt94" (UID: "3d0f4eb6-2bba-480e-8dae-1827c5ac731a") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:34.535563 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:34.535520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:34.535750 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.535736    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:34.535820 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.535761    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:34.535820 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.535774    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ntf9g for pod openshift-network-diagnostics/network-check-target-jk5dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:34.535944 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.535858    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g podName:caa455b9-a713-4cbb-8d32-26da115b04d1 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:36.535816812 +0000 UTC m=+6.241597501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntf9g" (UniqueName: "kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g") pod "network-check-target-jk5dm" (UID: "caa455b9-a713-4cbb-8d32-26da115b04d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:34.754020 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:34.753971    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 11:17:33 +0000 UTC" deadline="2028-04-15 09:55:08.601906361 +0000 UTC"
Sep 07 11:22:34.754020 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:34.754014    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14062h32m33.84789709s"
Sep 07 11:22:34.935136 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:34.935099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:34.935136 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:34.935134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:34.945704 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.935682    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:34.945704 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.943002    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:34.945704 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:34.943990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:34.945704 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:34.944141    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:35.975163 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:35.974252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" event={"ID":"20ada389393df0af1b24f49af160350a","Type":"ContainerStarted","Data":"0412f5ff58c2c241b35ca088228ae8a02fb6a46b08ebe6ea7b5648b3694bfad0"}
Sep 07 11:22:35.996074 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:35.995528    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" podStartSLOduration=3.995506913 podStartE2EDuration="3.995506913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:35.994683643 +0000 UTC m=+5.700464339" watchObservedRunningTime="2026-09-07 11:22:35.995506913 +0000 UTC m=+5.701287608"
Sep 07 11:22:36.357241 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:36.357159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:36.357387 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.357365    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:36.357446 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.357434    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs podName:85e5bbff-0cc8-45cb-91b8-b106b0a5a978 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:40.357413879 +0000 UTC m=+10.063194558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs") pod "network-metrics-daemon-lmtxp" (UID: "85e5bbff-0cc8-45cb-91b8-b106b0a5a978") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:36.457777 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:36.457549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:36.457777 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.457732    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:36.458071 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.457794    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret podName:3d0f4eb6-2bba-480e-8dae-1827c5ac731a nodeName:}" failed. No retries permitted until 2026-09-07 11:22:40.457774251 +0000 UTC m=+10.163554926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret") pod "global-pull-secret-syncer-nkt94" (UID: "3d0f4eb6-2bba-480e-8dae-1827c5ac731a") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:36.558009 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:36.557969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:36.558212 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.558150    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:36.558212 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.558169    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:36.558212 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.558183    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ntf9g for pod openshift-network-diagnostics/network-check-target-jk5dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:36.558389 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.558244    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g podName:caa455b9-a713-4cbb-8d32-26da115b04d1 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:40.55822182 +0000 UTC m=+10.264002530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntf9g" (UniqueName: "kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g") pod "network-check-target-jk5dm" (UID: "caa455b9-a713-4cbb-8d32-26da115b04d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:36.937956 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:36.937920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:36.938157 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.938062    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:36.938157 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:36.938146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:36.938288 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.938253    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:36.938348 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:36.938295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:36.938402 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:36.938360    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:36.982600 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:36.982562    2581 generic.go:356] "Generic (PLEG): container finished" podID="9a6999f592201b6868f277532693fdb1" containerID="85224c6cb7a4e675b683e8954333db0c0461e73eedd0484cf709ddfcbe62fced" exitCode=0
Sep 07 11:22:36.983535 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:36.983504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"9a6999f592201b6868f277532693fdb1","Type":"ContainerDied","Data":"85224c6cb7a4e675b683e8954333db0c0461e73eedd0484cf709ddfcbe62fced"}
Sep 07 11:22:38.937955 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:38.937919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:38.938438 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:38.938087    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:38.938438 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:38.938161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:38.938438 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:38.938280    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:38.938438 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:38.938339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:38.938438 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:38.938407    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:40.392366 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:40.392326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:40.392843 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.392517    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:40.392843 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.392586    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs podName:85e5bbff-0cc8-45cb-91b8-b106b0a5a978 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:48.392566462 +0000 UTC m=+18.098347148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs") pod "network-metrics-daemon-lmtxp" (UID: "85e5bbff-0cc8-45cb-91b8-b106b0a5a978") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:40.493912 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:40.493624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:40.493912 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.493870    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:40.494152 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.493940    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret podName:3d0f4eb6-2bba-480e-8dae-1827c5ac731a nodeName:}" failed. No retries permitted until 2026-09-07 11:22:48.493920579 +0000 UTC m=+18.199701248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret") pod "global-pull-secret-syncer-nkt94" (UID: "3d0f4eb6-2bba-480e-8dae-1827c5ac731a") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:40.594175 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:40.594041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:40.594357 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.594212    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:40.594357 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.594239    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:40.594357 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.594253    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ntf9g for pod openshift-network-diagnostics/network-check-target-jk5dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:40.594357 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.594320    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g podName:caa455b9-a713-4cbb-8d32-26da115b04d1 nodeName:}" failed. No retries permitted until 2026-09-07 11:22:48.594299939 +0000 UTC m=+18.300080613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntf9g" (UniqueName: "kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g") pod "network-check-target-jk5dm" (UID: "caa455b9-a713-4cbb-8d32-26da115b04d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:40.937076 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:40.936385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:40.937076 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.936497    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:40.937076 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:40.936933    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:40.937076 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.937030    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:40.937726 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:40.937495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:40.937726 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:40.937687    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:42.938836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:42.937991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:42.938836 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:42.938117    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:42.938836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:42.938496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:42.938836 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:42.938635    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:42.938836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:42.938696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:42.938836 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:42.938772    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:44.935249 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:44.935213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:44.935663 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:44.935215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:44.935663 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:44.935335    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:44.935663 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:44.935225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:44.935663 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:44.935417    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:44.935663 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:44.935460    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:46.935173 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:46.935115    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:46.935653 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:46.935140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:46.935653 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:46.935309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:46.935653 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:46.935363    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:46.935653 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:46.935116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:46.935653 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:46.935456    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:48.455929 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:48.455890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:48.456353 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.456043    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:48.456353 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.456116    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs podName:85e5bbff-0cc8-45cb-91b8-b106b0a5a978 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:04.456094023 +0000 UTC m=+34.161874713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs") pod "network-metrics-daemon-lmtxp" (UID: "85e5bbff-0cc8-45cb-91b8-b106b0a5a978") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:22:48.557294 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:48.557264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:48.557444 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.557389    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:48.557444 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.557439    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret podName:3d0f4eb6-2bba-480e-8dae-1827c5ac731a nodeName:}" failed. No retries permitted until 2026-09-07 11:23:04.557426461 +0000 UTC m=+34.263207142 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret") pod "global-pull-secret-syncer-nkt94" (UID: "3d0f4eb6-2bba-480e-8dae-1827c5ac731a") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:22:48.658530 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:48.658492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:48.658689 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.658674    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:22:48.658757 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.658699    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:22:48.658757 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.658708    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ntf9g for pod openshift-network-diagnostics/network-check-target-jk5dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:48.658871 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.658766    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g podName:caa455b9-a713-4cbb-8d32-26da115b04d1 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:04.658750504 +0000 UTC m=+34.364531192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntf9g" (UniqueName: "kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g") pod "network-check-target-jk5dm" (UID: "caa455b9-a713-4cbb-8d32-26da115b04d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:22:48.934891 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:48.934843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:48.935078 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:48.934857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:48.935078 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.934972    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:48.935078 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:48.934855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:48.935208 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.935079    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:48.935208 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:48.935169    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:50.935377 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:50.935348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:50.935841 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:50.935442    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:50.935841 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:50.935512    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:50.935841 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:50.935598    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:50.935841 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:50.935644    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:50.935841 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:50.935705    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:52.935984 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:52.935617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:52.936513 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:52.935669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:52.936513 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:52.935669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:52.936513 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:52.936128    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:52.936513 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:52.936246    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:52.936513 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:52.936347    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:53.023437 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.023402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xpwmp" event={"ID":"43925274-591e-4a1c-972f-aba80b98abbf","Type":"ContainerStarted","Data":"fd8d1a5df4fef4926b89875131ae9815f44bd6be5fd1f12b257c8c0f5c95c0de"}
Sep 07 11:22:53.025059 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.025034    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1a3c173-d443-479b-9521-edf01ab367e8" containerID="0453a4fb3effdb755dcb776168ab8b93dba540de5153ef6ce8f9d5c91ed402fc" exitCode=0
Sep 07 11:22:53.025174 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.025108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" event={"ID":"b1a3c173-d443-479b-9521-edf01ab367e8","Type":"ContainerDied","Data":"0453a4fb3effdb755dcb776168ab8b93dba540de5153ef6ce8f9d5c91ed402fc"}
Sep 07 11:22:53.029491 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.027223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z4xbr" event={"ID":"40b2c47e-d219-4889-b4ed-d78eaa2f179e","Type":"ContainerStarted","Data":"32465f7bb673b2230e646f251126fa7acd1e1116e20233792a3fbac19b4c9f10"}
Sep 07 11:22:53.031088 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.031057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lt54v" event={"ID":"52a43c8b-4eaf-4514-a60f-d0fba1696bd0","Type":"ContainerStarted","Data":"8914f9d8e6fe6d262f0ab127f6bd76eab375cf8af45688107f9f88665982509b"}
Sep 07 11:22:53.032448 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.032429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl" event={"ID":"0db155b2-82b9-471a-b334-4c7b1ee71288","Type":"ContainerStarted","Data":"dc9a5b2e972219d4f75a6b83ffe05b1139ac57e1dc032330033f6fd230289c95"}
Sep 07 11:22:53.033949 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.033925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7w9gb" event={"ID":"0427ce89-4926-4887-a9fe-6d49fb517836","Type":"ContainerStarted","Data":"804cbcdf22820fa07cd559ba0d87a2ecfc709020188dc8a7db72f140efbf293b"}
Sep 07 11:22:53.035999 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.035974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"9a6999f592201b6868f277532693fdb1","Type":"ContainerStarted","Data":"9ebb2dbf809576c61ecf6e9c8496d672f37c64f643edc5f30a9022da66fb6e79"}
Sep 07 11:22:53.037457 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.037426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m8j27" event={"ID":"dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1","Type":"ContainerStarted","Data":"d33a7e79273780e576c43ce401f7d12341492a1f95a20a259caaee48b0d5f1f0"}
Sep 07 11:22:53.039592 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.039575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"d564610ea887d432a2831eb9af06596baa581240bef7f4e42d3b2c81f0139bb8"}
Sep 07 11:22:53.039663 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.039629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"b775ab491340dcfac1a2005426da1c9516b66c5feb5131a040fb99153a7feb55"}
Sep 07 11:22:53.039714 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.039664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"f35cf0a70770797a70fe2d758631b51a2b54e2f8a698577d8063bef670fedd10"}
Sep 07 11:22:53.042200 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.042158    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xpwmp" podStartSLOduration=1.5291880089999998 podStartE2EDuration="21.042145927s" totalImagesPullingTime="19.512957918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:53.042058139 +0000 UTC m=+22.747838828" watchObservedRunningTime="2026-09-07 11:22:53.042145927 +0000 UTC m=+22.747926617"
Sep 07 11:22:53.087496 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.087459    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" podStartSLOduration=21.087445448 podStartE2EDuration="21.087445448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:53.059266758 +0000 UTC m=+22.765047447" watchObservedRunningTime="2026-09-07 11:22:53.087445448 +0000 UTC m=+22.793226138"
Sep 07 11:22:53.087835 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.087811    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7w9gb" podStartSLOduration=2.576722858 podStartE2EDuration="22.087804917s" totalImagesPullingTime="19.511082059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:53.087316511 +0000 UTC m=+22.793097201" watchObservedRunningTime="2026-09-07 11:22:53.087804917 +0000 UTC m=+22.793585603"
Sep 07 11:22:53.128134 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.128094    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lt54v" podStartSLOduration=1.646635246 podStartE2EDuration="21.128079976s" totalImagesPullingTime="19.48144473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:53.127600222 +0000 UTC m=+22.833380911" watchObservedRunningTime="2026-09-07 11:22:53.128079976 +0000 UTC m=+22.833860701"
Sep 07 11:22:53.148077 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.148042    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m8j27" podStartSLOduration=1.6729681950000002 podStartE2EDuration="21.148027724s" totalImagesPullingTime="19.475059529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:53.147637126 +0000 UTC m=+22.853417818" watchObservedRunningTime="2026-09-07 11:22:53.148027724 +0000 UTC m=+22.853808414"
Sep 07 11:22:53.162902 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:53.162864    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z4xbr" podStartSLOduration=9.218986766 podStartE2EDuration="21.162849382s" totalImagesPullingTime="11.943862616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:53.162514796 +0000 UTC m=+22.868295486" watchObservedRunningTime="2026-09-07 11:22:53.162849382 +0000 UTC m=+22.868630072"
Sep 07 11:22:54.045690 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.045651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"df804052eb4aa85bd9244a218696b3f9b27d9405a440bfd71562b4718cc46a2e"}
Sep 07 11:22:54.046351 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.045747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"d116b8860eed8fa5a1a6adb8d0153fb9a15dbfb55f39d615c9a8b041040705d7"}
Sep 07 11:22:54.046351 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.045817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"4dcb74fcf2d7f48151a7618ba9e03de53e6c1d455351f8d9d76db9a4f5ac8084"}
Sep 07 11:22:54.622247 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.622217    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 11:22:54.920065 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.919862    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T11:22:54.622243535Z","UUID":"5198428f-b39a-4d2f-a271-5d9ed5a40abd","Handler":null,"Name":"","Endpoint":""}
Sep 07 11:22:54.921581 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.921561    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 11:22:54.921698 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.921590    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 11:22:54.935180 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.935156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:54.935316 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.935200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:54.935316 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:54.935156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:54.935316 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:54.935270    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:54.935480 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:54.935378    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:54.935536 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:54.935478    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:55.049400 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:55.049352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl" event={"ID":"0db155b2-82b9-471a-b334-4c7b1ee71288","Type":"ContainerStarted","Data":"07e30ca093c5931da870391dec060c581efde1057acea77559d7b3b5bbd11f5b"}
Sep 07 11:22:55.050707 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:55.050685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sh9c5" event={"ID":"3909660c-5e60-493f-b6fa-79ad570b4d90","Type":"ContainerStarted","Data":"5a6acff9740fe01be8c00c61426ef3a66319c0c908e14c36b9a0c2de432758fb"}
Sep 07 11:22:55.068210 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:55.068161    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sh9c5" podStartSLOduration=3.599023522 podStartE2EDuration="23.068143198s" totalImagesPullingTime="19.469119676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:55.067912288 +0000 UTC m=+24.773692979" watchObservedRunningTime="2026-09-07 11:22:55.068143198 +0000 UTC m=+24.773923889"
Sep 07 11:22:55.276142 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:55.276100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:55.276885 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:55.276862    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:56.057145 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:56.057103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"b15732366d5655e7414f3e5801d50c20c0597da411dd22fac7ba3aa465d6af1a"}
Sep 07 11:22:56.057711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:56.057312    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:56.057868 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:56.057852    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z4xbr"
Sep 07 11:22:56.938264 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:56.938235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:56.938264 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:56.938246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:56.938264 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:56.938246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:56.938548 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:56.938344    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:56.938548 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:56.938400    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:56.938548 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:56.938481    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:58.063474 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.063259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl" event={"ID":"0db155b2-82b9-471a-b334-4c7b1ee71288","Type":"ContainerStarted","Data":"219930a3f1ec70dd404c809c6c8b014871c62642bfba6ce2338e79535febc8b3"}
Sep 07 11:22:58.066984 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.066948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" event={"ID":"ab8aa650-17bb-4302-8c52-177600aae204","Type":"ContainerStarted","Data":"569b42a02e00dff8e535ae95c7c029ed344bd806d7302397628169be90f3fc00"}
Sep 07 11:22:58.067208 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.067182    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:58.067331 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.067260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:58.068958 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.068925    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1a3c173-d443-479b-9521-edf01ab367e8" containerID="53f44417e52df5f8b0b3a74ad5877ec99a255144b2ff09b77e26b6b919b2dc9e" exitCode=0
Sep 07 11:22:58.069038 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.069002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" event={"ID":"b1a3c173-d443-479b-9521-edf01ab367e8","Type":"ContainerDied","Data":"53f44417e52df5f8b0b3a74ad5877ec99a255144b2ff09b77e26b6b919b2dc9e"}
Sep 07 11:22:58.083194 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.083171    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:58.089070 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.089031    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmzsl" podStartSLOduration=2.9634364939999998 podStartE2EDuration="27.089017952s" totalImagesPullingTime="24.125581458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:58.087647167 +0000 UTC m=+27.793427904" watchObservedRunningTime="2026-09-07 11:22:58.089017952 +0000 UTC m=+27.794798679"
Sep 07 11:22:58.149602 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.149553    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z" podStartSLOduration=7.59214899 podStartE2EDuration="27.149539064s" totalImagesPullingTime="19.557390074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:22:58.148290389 +0000 UTC m=+27.854071079" watchObservedRunningTime="2026-09-07 11:22:58.149539064 +0000 UTC m=+27.855319762"
Sep 07 11:22:58.936999 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.936918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:58.937154 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:58.937065    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:22:58.937154 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.937088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:58.937154 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:58.937127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:58.937317 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:58.937230    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:58.937393 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:58.937312    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:59.076836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.076588    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1a3c173-d443-479b-9521-edf01ab367e8" containerID="07fb31a16d9b844ccf97ecee512f821453b7f531231dd7b1b8082cd8ba45f6da" exitCode=0
Sep 07 11:22:59.077228 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.076663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" event={"ID":"b1a3c173-d443-479b-9521-edf01ab367e8","Type":"ContainerDied","Data":"07fb31a16d9b844ccf97ecee512f821453b7f531231dd7b1b8082cd8ba45f6da"}
Sep 07 11:22:59.077558 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.077504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:59.092565 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.092545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:22:59.565039 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.565009    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jk5dm"]
Sep 07 11:22:59.565216 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.565125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:22:59.565298 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:59.565229    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:22:59.567454 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.567431    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lmtxp"]
Sep 07 11:22:59.567548 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.567528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:22:59.567649 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:59.567630    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:22:59.578619 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.578596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nkt94"]
Sep 07 11:22:59.578736 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:22:59.578678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:22:59.578832 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:22:59.578747    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:23:00.081002 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:00.080971    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1a3c173-d443-479b-9521-edf01ab367e8" containerID="eaa83cebbee9c19679c6ba0cc5447945a882a517ad64d3244fb252179e73bc21" exitCode=0
Sep 07 11:23:00.081572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:00.081057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" event={"ID":"b1a3c173-d443-479b-9521-edf01ab367e8","Type":"ContainerDied","Data":"eaa83cebbee9c19679c6ba0cc5447945a882a517ad64d3244fb252179e73bc21"}
Sep 07 11:23:00.940627 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:00.940595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:23:00.940812 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:00.940595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:00.940812 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:00.940704    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:23:00.940812 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:00.940763    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:23:01.935476 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:01.935404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:23:01.935934 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:01.935590    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:23:02.934828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:02.934780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:02.935005 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:02.934930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:23:02.935005 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:02.934980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:23:02.935110 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:02.935072    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:23:03.934950 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:03.934914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:23:03.935505 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:03.935034    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmtxp" podUID="85e5bbff-0cc8-45cb-91b8-b106b0a5a978"
Sep 07 11:23:04.472708 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:04.472673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:23:04.472908 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.472834    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:23:04.472974 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.472916    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs podName:85e5bbff-0cc8-45cb-91b8-b106b0a5a978 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:36.472880484 +0000 UTC m=+66.178661152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs") pod "network-metrics-daemon-lmtxp" (UID: "85e5bbff-0cc8-45cb-91b8-b106b0a5a978") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 11:23:04.573051 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:04.573012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:23:04.573245 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.573180    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 11:23:04.573308 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.573260    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret podName:3d0f4eb6-2bba-480e-8dae-1827c5ac731a nodeName:}" failed. No retries permitted until 2026-09-07 11:23:36.573235579 +0000 UTC m=+66.279016247 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret") pod "global-pull-secret-syncer-nkt94" (UID: "3d0f4eb6-2bba-480e-8dae-1827c5ac731a") : object "kube-system"/"original-pull-secret" not registered
Sep 07 11:23:04.674231 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:04.674194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:04.674404 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.674352    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 11:23:04.674404 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.674370    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 11:23:04.674404 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.674379    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ntf9g for pod openshift-network-diagnostics/network-check-target-jk5dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:23:04.674537 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.674446    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g podName:caa455b9-a713-4cbb-8d32-26da115b04d1 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:36.674429749 +0000 UTC m=+66.380210418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntf9g" (UniqueName: "kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g") pod "network-check-target-jk5dm" (UID: "caa455b9-a713-4cbb-8d32-26da115b04d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 11:23:04.935062 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:04.934977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:04.935531 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.935083    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk5dm" podUID="caa455b9-a713-4cbb-8d32-26da115b04d1"
Sep 07 11:23:04.935531 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:04.935140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:23:04.935531 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:04.935230    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkt94" podUID="3d0f4eb6-2bba-480e-8dae-1827c5ac731a"
Sep 07 11:23:05.557602 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.557483    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeReady"
Sep 07 11:23:05.557649 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.557613    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 11:23:05.590874 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.590817    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6968776c5f-zfzqj"]
Sep 07 11:23:05.608071 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.608045    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h9qtv"]
Sep 07 11:23:05.608202 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.608185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.610681 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.610657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 07 11:23:05.610818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.610785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 07 11:23:05.610963 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.610946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v974h\""
Sep 07 11:23:05.611008 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.610972    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 07 11:23:05.616515 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.616494    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 07 11:23:05.622225 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.622205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6968776c5f-zfzqj"]
Sep 07 11:23:05.622325 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.622233    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5crfb"]
Sep 07 11:23:05.622325 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.622314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.624232 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.624216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 11:23:05.624582 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.624565    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 11:23:05.624691 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.624680    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ph89j\""
Sep 07 11:23:05.640364 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.640332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h9qtv"]
Sep 07 11:23:05.640489 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.640375    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5crfb"]
Sep 07 11:23:05.640560 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.640498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5crfb"
Sep 07 11:23:05.642634 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.642616    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4j7np\""
Sep 07 11:23:05.642864 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.642844    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 11:23:05.642975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.642881    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 11:23:05.642975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.642914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 11:23:05.742192 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.742147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z4xbr_40b2c47e-d219-4889-b4ed-d78eaa2f179e/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:05.782777 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.782752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgc9f\" (UniqueName: \"kubernetes.io/projected/8aecb463-b248-47aa-8ac7-0e15b220c491-kube-api-access-bgc9f\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.782919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.782808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.782919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.782839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aecb463-b248-47aa-8ac7-0e15b220c491-config-volume\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.782919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.782864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-registry-certificates\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.782919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.782884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pscx2\" (UniqueName: \"kubernetes.io/projected/a74ec153-c250-4823-9bef-93811a97c81d-kube-api-access-pscx2\") pod \"ingress-canary-5crfb\" (UID: \"a74ec153-c250-4823-9bef-93811a97c81d\") " pod="openshift-ingress-canary/ingress-canary-5crfb"
Sep 07 11:23:05.782919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.782900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-bound-sa-token\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.783129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.782941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-installation-pull-secrets\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.783129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.782968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s49v\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-kube-api-access-9s49v\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.783129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.783008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a74ec153-c250-4823-9bef-93811a97c81d-cert\") pod \"ingress-canary-5crfb\" (UID: \"a74ec153-c250-4823-9bef-93811a97c81d\") " pod="openshift-ingress-canary/ingress-canary-5crfb"
Sep 07 11:23:05.783129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.783040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aecb463-b248-47aa-8ac7-0e15b220c491-tmp-dir\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.783129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.783055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-image-registry-private-configuration\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.783129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.783118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.783318 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.783149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/481046bf-a011-484e-8e20-18b2d7edc210-ca-trust-extracted\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.783318 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.783171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-trusted-ca\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.861811 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.861707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z4xbr_40b2c47e-d219-4889-b4ed-d78eaa2f179e/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:05.883734 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.883709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.883820 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.883748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aecb463-b248-47aa-8ac7-0e15b220c491-config-volume\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.883820 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.883767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-registry-certificates\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.883984 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:05.883965    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 11:23:05.884017 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.883981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pscx2\" (UniqueName: \"kubernetes.io/projected/a74ec153-c250-4823-9bef-93811a97c81d-kube-api-access-pscx2\") pod \"ingress-canary-5crfb\" (UID: \"a74ec153-c250-4823-9bef-93811a97c81d\") " pod="openshift-ingress-canary/ingress-canary-5crfb"
Sep 07 11:23:05.884062 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-bound-sa-token\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.884062 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:05.883988    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6968776c5f-zfzqj: secret "image-registry-tls" not found
Sep 07 11:23:05.884062 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-installation-pull-secrets\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.884200 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s49v\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-kube-api-access-9s49v\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.884200 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:05.884108    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls podName:481046bf-a011-484e-8e20-18b2d7edc210 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:06.384088592 +0000 UTC m=+36.089869260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls") pod "image-registry-6968776c5f-zfzqj" (UID: "481046bf-a011-484e-8e20-18b2d7edc210") : secret "image-registry-tls" not found
Sep 07 11:23:05.884200 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a74ec153-c250-4823-9bef-93811a97c81d-cert\") pod \"ingress-canary-5crfb\" (UID: \"a74ec153-c250-4823-9bef-93811a97c81d\") " pod="openshift-ingress-canary/ingress-canary-5crfb"
Sep 07 11:23:05.884200 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aecb463-b248-47aa-8ac7-0e15b220c491-tmp-dir\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.884200 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-image-registry-private-configuration\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.884436 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.884436 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/481046bf-a011-484e-8e20-18b2d7edc210-ca-trust-extracted\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.884436 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-trusted-ca\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.884436 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgc9f\" (UniqueName: \"kubernetes.io/projected/8aecb463-b248-47aa-8ac7-0e15b220c491-kube-api-access-bgc9f\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.884436 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-registry-certificates\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.884436 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:05.884369    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:05.884436 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aecb463-b248-47aa-8ac7-0e15b220c491-config-volume\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.884436 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:05.884416    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls podName:8aecb463-b248-47aa-8ac7-0e15b220c491 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:06.384400198 +0000 UTC m=+36.090180881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls") pod "dns-default-h9qtv" (UID: "8aecb463-b248-47aa-8ac7-0e15b220c491") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:05.884855 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/481046bf-a011-484e-8e20-18b2d7edc210-ca-trust-extracted\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.884855 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.884812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aecb463-b248-47aa-8ac7-0e15b220c491-tmp-dir\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.885474 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.885445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-trusted-ca\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.888074 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.888054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-image-registry-private-configuration\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.888074 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.888067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a74ec153-c250-4823-9bef-93811a97c81d-cert\") pod \"ingress-canary-5crfb\" (UID: \"a74ec153-c250-4823-9bef-93811a97c81d\") " pod="openshift-ingress-canary/ingress-canary-5crfb"
Sep 07 11:23:05.888196 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.888088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-installation-pull-secrets\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.892587 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.892557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pscx2\" (UniqueName: \"kubernetes.io/projected/a74ec153-c250-4823-9bef-93811a97c81d-kube-api-access-pscx2\") pod \"ingress-canary-5crfb\" (UID: \"a74ec153-c250-4823-9bef-93811a97c81d\") " pod="openshift-ingress-canary/ingress-canary-5crfb"
Sep 07 11:23:05.892910 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.892892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s49v\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-kube-api-access-9s49v\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.893047 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.893029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-bound-sa-token\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:05.893518 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.893500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgc9f\" (UniqueName: \"kubernetes.io/projected/8aecb463-b248-47aa-8ac7-0e15b220c491-kube-api-access-bgc9f\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:05.935522 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.935501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:23:05.937961 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.937947    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 11:23:05.938059 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.937972    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qlhnv\""
Sep 07 11:23:05.949207 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:05.949190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5crfb"
Sep 07 11:23:06.098688 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.098656    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1a3c173-d443-479b-9521-edf01ab367e8" containerID="7d9a4d7650c415a38f3e96d469e8812ac74cd9ad72013502f3294fc572f1a60e" exitCode=0
Sep 07 11:23:06.098843 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.098698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" event={"ID":"b1a3c173-d443-479b-9521-edf01ab367e8","Type":"ContainerDied","Data":"7d9a4d7650c415a38f3e96d469e8812ac74cd9ad72013502f3294fc572f1a60e"}
Sep 07 11:23:06.274692 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.274536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5crfb"]
Sep 07 11:23:06.278029 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:23:06.278008    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74ec153_c250_4823_9bef_93811a97c81d.slice/crio-b58e5aabe13ecf079a527dc0310e061ec5da8fccf0016dfe4d6a6429971da3a3 WatchSource:0}: Error finding container b58e5aabe13ecf079a527dc0310e061ec5da8fccf0016dfe4d6a6429971da3a3: Status 404 returned error can't find the container with id b58e5aabe13ecf079a527dc0310e061ec5da8fccf0016dfe4d6a6429971da3a3
Sep 07 11:23:06.387971 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.387896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:06.388083 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:06.388048    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 11:23:06.388083 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:06.388066    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6968776c5f-zfzqj: secret "image-registry-tls" not found
Sep 07 11:23:06.388168 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.388083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:06.388168 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:06.388112    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls podName:481046bf-a011-484e-8e20-18b2d7edc210 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:07.388098589 +0000 UTC m=+37.093879257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls") pod "image-registry-6968776c5f-zfzqj" (UID: "481046bf-a011-484e-8e20-18b2d7edc210") : secret "image-registry-tls" not found
Sep 07 11:23:06.388168 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:06.388166    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:06.388270 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:06.388211    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls podName:8aecb463-b248-47aa-8ac7-0e15b220c491 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:07.38819896 +0000 UTC m=+37.093979627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls") pod "dns-default-h9qtv" (UID: "8aecb463-b248-47aa-8ac7-0e15b220c491") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:06.935128 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.935089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:23:06.935316 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.935155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:06.937712 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.937691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 11:23:06.938126 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.937760    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 11:23:06.938454 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.938427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8r9c\""
Sep 07 11:23:06.938454 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:06.938453    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 11:23:07.102950 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:07.102913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5crfb" event={"ID":"a74ec153-c250-4823-9bef-93811a97c81d","Type":"ContainerStarted","Data":"b58e5aabe13ecf079a527dc0310e061ec5da8fccf0016dfe4d6a6429971da3a3"}
Sep 07 11:23:07.106604 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:07.106576    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1a3c173-d443-479b-9521-edf01ab367e8" containerID="88f78c13f0bc537edcaf1828b2eabc60fef9ff02ebb0553e1f61c1f021125c77" exitCode=0
Sep 07 11:23:07.106749 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:07.106615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" event={"ID":"b1a3c173-d443-479b-9521-edf01ab367e8","Type":"ContainerDied","Data":"88f78c13f0bc537edcaf1828b2eabc60fef9ff02ebb0553e1f61c1f021125c77"}
Sep 07 11:23:07.396868 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:07.396829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:07.397053 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:07.396890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:07.397053 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:07.397002    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 11:23:07.397053 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:07.397028    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6968776c5f-zfzqj: secret "image-registry-tls" not found
Sep 07 11:23:07.397053 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:07.397003    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:07.397215 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:07.397095    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls podName:481046bf-a011-484e-8e20-18b2d7edc210 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:09.397075565 +0000 UTC m=+39.102856239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls") pod "image-registry-6968776c5f-zfzqj" (UID: "481046bf-a011-484e-8e20-18b2d7edc210") : secret "image-registry-tls" not found
Sep 07 11:23:07.397215 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:07.397112    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls podName:8aecb463-b248-47aa-8ac7-0e15b220c491 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:09.397102592 +0000 UTC m=+39.102883264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls") pod "dns-default-h9qtv" (UID: "8aecb463-b248-47aa-8ac7-0e15b220c491") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:08.113005 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:08.112966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" event={"ID":"b1a3c173-d443-479b-9521-edf01ab367e8","Type":"ContainerStarted","Data":"dbdc74eff35862cc7a2d05e98e6ddc7709cac2cb0f120a614275b650520ffb85"}
Sep 07 11:23:08.135489 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:08.135435    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9hpnc" podStartSLOduration=5.283692688 podStartE2EDuration="36.135418471s" totalImagesPullingTime="30.851725783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:08.134743296 +0000 UTC m=+37.840524012" watchObservedRunningTime="2026-09-07 11:23:08.135418471 +0000 UTC m=+37.841199160"
Sep 07 11:23:08.254204 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:08.254168    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z4xbr_40b2c47e-d219-4889-b4ed-d78eaa2f179e/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:08.314712 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:08.314654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z4xbr_40b2c47e-d219-4889-b4ed-d78eaa2f179e/konnectivity-agent/0.log" limit=1024
Sep 07 11:23:09.116828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:09.116775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5crfb" event={"ID":"a74ec153-c250-4823-9bef-93811a97c81d","Type":"ContainerStarted","Data":"85875a40eabfc09040bf60179ec69950b74e41af612257be579ef8ed8307beff"}
Sep 07 11:23:09.132229 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:09.132176    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5crfb" podStartSLOduration=1.474360395 podStartE2EDuration="4.132160871s" totalImagesPullingTime="2.657800476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:09.131639831 +0000 UTC m=+38.837420522" watchObservedRunningTime="2026-09-07 11:23:09.132160871 +0000 UTC m=+38.837941582"
Sep 07 11:23:09.415527 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:09.415491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:09.415692 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:09.415566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:09.415692 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:09.415660    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:09.415692 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:09.415672    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 11:23:09.415822 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:09.415695    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6968776c5f-zfzqj: secret "image-registry-tls" not found
Sep 07 11:23:09.415822 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:09.415712    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls podName:8aecb463-b248-47aa-8ac7-0e15b220c491 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:13.41569866 +0000 UTC m=+43.121479328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls") pod "dns-default-h9qtv" (UID: "8aecb463-b248-47aa-8ac7-0e15b220c491") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:09.415822 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:09.415749    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls podName:481046bf-a011-484e-8e20-18b2d7edc210 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:13.415730993 +0000 UTC m=+43.121511684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls") pod "image-registry-6968776c5f-zfzqj" (UID: "481046bf-a011-484e-8e20-18b2d7edc210") : secret "image-registry-tls" not found
Sep 07 11:23:13.443528 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:13.443490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:13.444102 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:13.443549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:13.444102 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:13.443635    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 11:23:13.444102 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:13.443646    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6968776c5f-zfzqj: secret "image-registry-tls" not found
Sep 07 11:23:13.444102 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:13.443656    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:13.444102 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:13.443698    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls podName:481046bf-a011-484e-8e20-18b2d7edc210 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:21.443684612 +0000 UTC m=+51.149465279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls") pod "image-registry-6968776c5f-zfzqj" (UID: "481046bf-a011-484e-8e20-18b2d7edc210") : secret "image-registry-tls" not found
Sep 07 11:23:13.444102 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:13.443722    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls podName:8aecb463-b248-47aa-8ac7-0e15b220c491 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:21.443702568 +0000 UTC m=+51.149483237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls") pod "dns-default-h9qtv" (UID: "8aecb463-b248-47aa-8ac7-0e15b220c491") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:21.502697 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:21.502654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:21.503167 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:21.502733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:21.503167 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:21.502804    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 11:23:21.503167 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:21.502824    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6968776c5f-zfzqj: secret "image-registry-tls" not found
Sep 07 11:23:21.503167 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:21.502830    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:21.503167 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:21.502879    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls podName:8aecb463-b248-47aa-8ac7-0e15b220c491 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:37.502864733 +0000 UTC m=+67.208645400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls") pod "dns-default-h9qtv" (UID: "8aecb463-b248-47aa-8ac7-0e15b220c491") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:21.503167 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:21.502893    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls podName:481046bf-a011-484e-8e20-18b2d7edc210 nodeName:}" failed. No retries permitted until 2026-09-07 11:23:37.502885715 +0000 UTC m=+67.208666383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls") pod "image-registry-6968776c5f-zfzqj" (UID: "481046bf-a011-484e-8e20-18b2d7edc210") : secret "image-registry-tls" not found
Sep 07 11:23:31.093862 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:31.093832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-prn2z"
Sep 07 11:23:36.511058 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.511012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:23:36.513611 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.513586    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 11:23:36.521747 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:36.521725    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 07 11:23:36.521812 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:36.521785    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs podName:85e5bbff-0cc8-45cb-91b8-b106b0a5a978 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:40.52176896 +0000 UTC m=+130.227549628 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs") pod "network-metrics-daemon-lmtxp" (UID: "85e5bbff-0cc8-45cb-91b8-b106b0a5a978") : secret "metrics-daemon-secret" not found
Sep 07 11:23:36.612148 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.612119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:23:36.614872 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.614856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 11:23:36.625826 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.625802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d0f4eb6-2bba-480e-8dae-1827c5ac731a-original-pull-secret\") pod \"global-pull-secret-syncer-nkt94\" (UID: \"3d0f4eb6-2bba-480e-8dae-1827c5ac731a\") " pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:23:36.647508 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.647487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkt94"
Sep 07 11:23:36.713100 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.713066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:36.715837 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.715814    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 11:23:36.726253 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.726225    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 11:23:36.737035 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.737011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntf9g\" (UniqueName: \"kubernetes.io/projected/caa455b9-a713-4cbb-8d32-26da115b04d1-kube-api-access-ntf9g\") pod \"network-check-target-jk5dm\" (UID: \"caa455b9-a713-4cbb-8d32-26da115b04d1\") " pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:36.944034 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.943968    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nkt94"]
Sep 07 11:23:36.947089 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:23:36.947063    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0f4eb6_2bba_480e_8dae_1827c5ac731a.slice/crio-57191c3218457f9809fd08a0e30f2913909ddf767ac1cc9307647d76f0599ecf WatchSource:0}: Error finding container 57191c3218457f9809fd08a0e30f2913909ddf767ac1cc9307647d76f0599ecf: Status 404 returned error can't find the container with id 57191c3218457f9809fd08a0e30f2913909ddf767ac1cc9307647d76f0599ecf
Sep 07 11:23:36.954557 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.954536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8r9c\""
Sep 07 11:23:36.963259 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:36.963237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:37.177905 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:37.177874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nkt94" event={"ID":"3d0f4eb6-2bba-480e-8dae-1827c5ac731a","Type":"ContainerStarted","Data":"57191c3218457f9809fd08a0e30f2913909ddf767ac1cc9307647d76f0599ecf"}
Sep 07 11:23:37.228485 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:37.228457    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jk5dm"]
Sep 07 11:23:37.231560 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:23:37.231527    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa455b9_a713_4cbb_8d32_26da115b04d1.slice/crio-e7395f3d704f4ad9676d47d579ec4be1d03503ddbc686ffde5efad1eb0a631a9 WatchSource:0}: Error finding container e7395f3d704f4ad9676d47d579ec4be1d03503ddbc686ffde5efad1eb0a631a9: Status 404 returned error can't find the container with id e7395f3d704f4ad9676d47d579ec4be1d03503ddbc686ffde5efad1eb0a631a9
Sep 07 11:23:37.519391 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:37.519304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:23:37.519391 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:37.519389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:23:37.519954 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:37.519500    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 11:23:37.519954 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:37.519532    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6968776c5f-zfzqj: secret "image-registry-tls" not found
Sep 07 11:23:37.519954 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:37.519503    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:23:37.519954 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:37.519599    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls podName:8aecb463-b248-47aa-8ac7-0e15b220c491 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.519577574 +0000 UTC m=+99.225358246 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls") pod "dns-default-h9qtv" (UID: "8aecb463-b248-47aa-8ac7-0e15b220c491") : secret "dns-default-metrics-tls" not found
Sep 07 11:23:37.519954 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:23:37.519618    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls podName:481046bf-a011-484e-8e20-18b2d7edc210 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.519608095 +0000 UTC m=+99.225388770 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls") pod "image-registry-6968776c5f-zfzqj" (UID: "481046bf-a011-484e-8e20-18b2d7edc210") : secret "image-registry-tls" not found
Sep 07 11:23:38.182703 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:38.182638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jk5dm" event={"ID":"caa455b9-a713-4cbb-8d32-26da115b04d1","Type":"ContainerStarted","Data":"e7395f3d704f4ad9676d47d579ec4be1d03503ddbc686ffde5efad1eb0a631a9"}
Sep 07 11:23:42.195418 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:42.195333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nkt94" event={"ID":"3d0f4eb6-2bba-480e-8dae-1827c5ac731a","Type":"ContainerStarted","Data":"6b2516dd78a987d0ca73bd04feb7d2608f0b6b0f813e237c449b59ea64d4e276"}
Sep 07 11:23:42.196801 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:42.196762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jk5dm" event={"ID":"caa455b9-a713-4cbb-8d32-26da115b04d1","Type":"ContainerStarted","Data":"0a9e4221ad2c579ebda3f695a1669080cefbf6c9c4828a6d5b0d4ca27d831afb"}
Sep 07 11:23:42.196926 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:42.196906    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:23:42.211008 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:42.210964    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nkt94" podStartSLOduration=65.412635265 podStartE2EDuration="1m10.210951822s" totalImagesPullingTime="4.798316557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:42.210254231 +0000 UTC m=+71.916034922" watchObservedRunningTime="2026-09-07 11:23:42.210951822 +0000 UTC m=+71.916732512"
Sep 07 11:23:42.224447 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:23:42.224407    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jk5dm" podStartSLOduration=65.709935874 podStartE2EDuration="1m10.224395771s" totalImagesPullingTime="4.514459897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:23:42.224092497 +0000 UTC m=+71.929873189" watchObservedRunningTime="2026-09-07 11:23:42.224395771 +0000 UTC m=+71.930176454"
Sep 07 11:24:01.990412 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.990299    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5895868fb6-s4mnj"]
Sep 07 11:24:01.994476 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.994456    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:01.996750 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.996723    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 07 11:24:01.996962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.996947    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 07 11:24:01.997027 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.997010    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 11:24:01.997069 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.997034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 07 11:24:01.997155 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.997139    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 11:24:01.997248 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.997230    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 07 11:24:01.997645 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:01.997627    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hx8p6\""
Sep 07 11:24:02.008908 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.008887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5895868fb6-s4mnj"]
Sep 07 11:24:02.094169 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.094135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.094169 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.094170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.094346 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.094207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-stats-auth\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.094346 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.094252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7h8\" (UniqueName: \"kubernetes.io/projected/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-kube-api-access-fs7h8\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.094346 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.094299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-default-certificate\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.194695 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.194663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7h8\" (UniqueName: \"kubernetes.io/projected/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-kube-api-access-fs7h8\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.194695 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.194696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-default-certificate\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.194887 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.194856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.194887 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.194883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.194961 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.194945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-stats-auth\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.195052 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:02.195033    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:02.695012747 +0000 UTC m=+92.400793428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : configmap references non-existent config key: service-ca.crt
Sep 07 11:24:02.195108 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:02.195045    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 11:24:02.195108 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:02.195096    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:02.695082774 +0000 UTC m=+92.400863442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : secret "router-metrics-certs-default" not found
Sep 07 11:24:02.197209 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.197187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-default-certificate\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.197304 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.197188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-stats-auth\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.204527 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.204507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7h8\" (UniqueName: \"kubernetes.io/projected/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-kube-api-access-fs7h8\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.698118 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.698067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.698118 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:02.698119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:02.698328 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:02.698261    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:03.698240203 +0000 UTC m=+93.404020888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : configmap references non-existent config key: service-ca.crt
Sep 07 11:24:02.698328 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:02.698262    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 11:24:02.698328 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:02.698302    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:03.698293151 +0000 UTC m=+93.404073823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : secret "router-metrics-certs-default" not found
Sep 07 11:24:03.706489 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:03.706452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:03.706489 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:03.706486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:03.706919 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:03.706597    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 11:24:03.706919 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:03.706626    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:05.706605242 +0000 UTC m=+95.412385910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : configmap references non-existent config key: service-ca.crt
Sep 07 11:24:03.706919 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:03.706649    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:05.706642733 +0000 UTC m=+95.412423401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : secret "router-metrics-certs-default" not found
Sep 07 11:24:05.721934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:05.721895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:05.721934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:05.721934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:05.722323 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:05.722037    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 11:24:05.722323 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:05.722093    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.722073455 +0000 UTC m=+99.427854129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : configmap references non-existent config key: service-ca.crt
Sep 07 11:24:05.722323 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:05.722112    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.72210493 +0000 UTC m=+99.427885597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : secret "router-metrics-certs-default" not found
Sep 07 11:24:07.381155 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.381124    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zhkmc"]
Sep 07 11:24:07.384202 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.384183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.386744 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.386723    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 11:24:07.387048 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.387021    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 11:24:07.387152 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.387073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 11:24:07.387830 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.387809    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6gln4\""
Sep 07 11:24:07.387939 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.387853    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 11:24:07.396093 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.396074    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhkmc"]
Sep 07 11:24:07.534957 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.534916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93118025-9b9a-4565-ab00-2de67090cf52-crio-socket\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.534957 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.534958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.535181 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.534987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93118025-9b9a-4565-ab00-2de67090cf52-data-volume\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.535181 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.535020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.535181 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.535061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqqx\" (UniqueName: \"kubernetes.io/projected/93118025-9b9a-4565-ab00-2de67090cf52-kube-api-access-5dqqx\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.636233 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.636170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93118025-9b9a-4565-ab00-2de67090cf52-data-volume\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.636233 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.636210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.636369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.636251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqqx\" (UniqueName: \"kubernetes.io/projected/93118025-9b9a-4565-ab00-2de67090cf52-kube-api-access-5dqqx\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.636369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.636306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93118025-9b9a-4565-ab00-2de67090cf52-crio-socket\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.636369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.636325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.636510 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:07.636395    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:07.636510 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.636410    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93118025-9b9a-4565-ab00-2de67090cf52-crio-socket\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.636510 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:07.636402    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:07.636510 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:07.636452    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:08.136436713 +0000 UTC m=+97.842217386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:07.636510 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:07.636473    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:08.136460101 +0000 UTC m=+97.842240774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:07.636510 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.636478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93118025-9b9a-4565-ab00-2de67090cf52-data-volume\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:07.644263 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:07.644232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqqx\" (UniqueName: \"kubernetes.io/projected/93118025-9b9a-4565-ab00-2de67090cf52-kube-api-access-5dqqx\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:08.139730 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:08.139695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:08.139900 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:08.139738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:08.139900 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:08.139826    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:08.139900 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:08.139883    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:08.139900 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:08.139893    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.139878166 +0000 UTC m=+98.845658835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:08.140058 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:08.139935    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:09.13992348 +0000 UTC m=+98.845704147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:09.146422 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:09.146391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:09.146776 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:09.146436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:09.146776 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.146519    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:09.146776 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.146534    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:09.146776 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.146587    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:11.146573917 +0000 UTC m=+100.852354599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:09.146776 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.146612    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:11.146591608 +0000 UTC m=+100.852372282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:09.532836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:09.532808    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m8j27_dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1/dns-node-resolver/0.log"
Sep 07 11:24:09.550242 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:09.550215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:24:09.550350 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:09.550257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") pod \"image-registry-6968776c5f-zfzqj\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") " pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:24:09.550410 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.550349    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 11:24:09.550410 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.550360    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6968776c5f-zfzqj: secret "image-registry-tls" not found
Sep 07 11:24:09.550410 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.550380    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 11:24:09.550410 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.550406    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls podName:481046bf-a011-484e-8e20-18b2d7edc210 nodeName:}" failed. No retries permitted until 2026-09-07 11:25:13.550393948 +0000 UTC m=+163.256174616 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls") pod "image-registry-6968776c5f-zfzqj" (UID: "481046bf-a011-484e-8e20-18b2d7edc210") : secret "image-registry-tls" not found
Sep 07 11:24:09.550570 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.550440    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls podName:8aecb463-b248-47aa-8ac7-0e15b220c491 nodeName:}" failed. No retries permitted until 2026-09-07 11:25:13.550421907 +0000 UTC m=+163.256202579 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls") pod "dns-default-h9qtv" (UID: "8aecb463-b248-47aa-8ac7-0e15b220c491") : secret "dns-default-metrics-tls" not found
Sep 07 11:24:09.752302 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:09.752264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:09.752302 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:09.752302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:09.752491 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.752432    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 11:24:09.752491 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.752433    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:17.752415912 +0000 UTC m=+107.458196601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : configmap references non-existent config key: service-ca.crt
Sep 07 11:24:09.752491 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:09.752477    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs podName:ffc78f96-d803-4e3e-bf08-a96f5b7e2aab nodeName:}" failed. No retries permitted until 2026-09-07 11:24:17.752466531 +0000 UTC m=+107.458247206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs") pod "router-default-5895868fb6-s4mnj" (UID: "ffc78f96-d803-4e3e-bf08-a96f5b7e2aab") : secret "router-metrics-certs-default" not found
Sep 07 11:24:10.133209 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:10.133179    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lt54v_52a43c8b-4eaf-4514-a60f-d0fba1696bd0/node-ca/0.log"
Sep 07 11:24:11.133955 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:11.133915    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5crfb_a74ec153-c250-4823-9bef-93811a97c81d/serve-healthcheck-canary/0.log"
Sep 07 11:24:11.162754 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:11.162723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:11.162903 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:11.162773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:11.162903 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:11.162852    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:11.162971 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:11.162905    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:11.162971 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:11.162911    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:15.162896976 +0000 UTC m=+104.868677648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:11.162971 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:11.162957    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:15.162946139 +0000 UTC m=+104.868726807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:13.202131 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:13.202100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jk5dm"
Sep 07 11:24:15.196254 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:15.196221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:15.196609 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:15.196389    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:15.196609 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:15.196406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:15.196609 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:15.196451    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:23.196436953 +0000 UTC m=+112.902217622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : secret "insights-runtime-extractor-tls" not found
Sep 07 11:24:15.196609 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:15.196490    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 11:24:15.196609 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:15.196533    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm podName:93118025-9b9a-4565-ab00-2de67090cf52 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:23.196522255 +0000 UTC m=+112.902302923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhkmc" (UID: "93118025-9b9a-4565-ab00-2de67090cf52") : configmap "kube-rbac-proxy" not found
Sep 07 11:24:17.817155 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:17.817119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:17.817155 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:17.817154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:17.817763 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:17.817740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-service-ca-bundle\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:17.819587 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:17.819558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc78f96-d803-4e3e-bf08-a96f5b7e2aab-metrics-certs\") pod \"router-default-5895868fb6-s4mnj\" (UID: \"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab\") " pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:17.903558 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:17.903525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:18.174601 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:18.174535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5895868fb6-s4mnj"]
Sep 07 11:24:18.177471 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:24:18.177438    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc78f96_d803_4e3e_bf08_a96f5b7e2aab.slice/crio-ae0f2e9f0008ab07b43bd2cb4dbb916fe9986bd56777a0f58e3ed3174a9f1cfb WatchSource:0}: Error finding container ae0f2e9f0008ab07b43bd2cb4dbb916fe9986bd56777a0f58e3ed3174a9f1cfb: Status 404 returned error can't find the container with id ae0f2e9f0008ab07b43bd2cb4dbb916fe9986bd56777a0f58e3ed3174a9f1cfb
Sep 07 11:24:18.285625 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:18.285589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5895868fb6-s4mnj" event={"ID":"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab","Type":"ContainerStarted","Data":"84358bb74150de18030c6fd8ff14d32e5c6cb5650833d4123581de746378ea89"}
Sep 07 11:24:18.285782 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:18.285642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5895868fb6-s4mnj" event={"ID":"ffc78f96-d803-4e3e-bf08-a96f5b7e2aab","Type":"ContainerStarted","Data":"ae0f2e9f0008ab07b43bd2cb4dbb916fe9986bd56777a0f58e3ed3174a9f1cfb"}
Sep 07 11:24:18.305300 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:18.305256    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5895868fb6-s4mnj" podStartSLOduration=17.305241855 podStartE2EDuration="17.305241855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:18.304502533 +0000 UTC m=+108.010283223" watchObservedRunningTime="2026-09-07 11:24:18.305241855 +0000 UTC m=+108.011022546"
Sep 07 11:24:18.904040 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:18.904005    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:18.906562 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:18.906537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:19.287714 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:19.287678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:19.288820 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:19.288783    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5895868fb6-s4mnj"
Sep 07 11:24:23.258978 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:23.258944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:23.259335 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:23.258988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:23.259453 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:23.259431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93118025-9b9a-4565-ab00-2de67090cf52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:23.261240 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:23.261220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93118025-9b9a-4565-ab00-2de67090cf52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhkmc\" (UID: \"93118025-9b9a-4565-ab00-2de67090cf52\") " pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:23.293235 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:23.293209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhkmc"
Sep 07 11:24:23.562200 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:23.562123    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhkmc"]
Sep 07 11:24:23.565089 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:24:23.565058    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93118025_9b9a_4565_ab00_2de67090cf52.slice/crio-deaf04d1c3e4e46ebec7cef40efec32e88a2e9c4ce3a95e5f0336bda88b3d3c6 WatchSource:0}: Error finding container deaf04d1c3e4e46ebec7cef40efec32e88a2e9c4ce3a95e5f0336bda88b3d3c6: Status 404 returned error can't find the container with id deaf04d1c3e4e46ebec7cef40efec32e88a2e9c4ce3a95e5f0336bda88b3d3c6
Sep 07 11:24:24.301891 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:24.301864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhkmc" event={"ID":"93118025-9b9a-4565-ab00-2de67090cf52","Type":"ContainerStarted","Data":"bcadba9f695657965478692aaaf32a012b00c90cc17e3f37daf344db2f304118"}
Sep 07 11:24:24.302152 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:24.301938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhkmc" event={"ID":"93118025-9b9a-4565-ab00-2de67090cf52","Type":"ContainerStarted","Data":"deaf04d1c3e4e46ebec7cef40efec32e88a2e9c4ce3a95e5f0336bda88b3d3c6"}
Sep 07 11:24:25.308031 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:25.308001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhkmc" event={"ID":"93118025-9b9a-4565-ab00-2de67090cf52","Type":"ContainerStarted","Data":"be315aeffafeae9b2ed34faaf4b5bfe9f68628b674f2100609b78a3adaeb71b3"}
Sep 07 11:24:26.312877 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:26.312839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhkmc" event={"ID":"93118025-9b9a-4565-ab00-2de67090cf52","Type":"ContainerStarted","Data":"ed5f768803041017802adf03aa152b9f6e133c9a3dc8e83d04e670939b50b8c0"}
Sep 07 11:24:26.330162 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:26.330119    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zhkmc" podStartSLOduration=17.322340119 podStartE2EDuration="19.330105007s" totalImagesPullingTime="2.007764888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:24:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:26.328878852 +0000 UTC m=+116.034659542" watchObservedRunningTime="2026-09-07 11:24:26.330105007 +0000 UTC m=+116.035885696"
Sep 07 11:24:29.765182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.765155    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"]
Sep 07 11:24:29.768227 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.768209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.771651 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.771623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 07 11:24:29.772464 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.772433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 07 11:24:29.772597 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.772577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 07 11:24:29.772812 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.772454    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 11:24:29.775118 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.774028    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 11:24:29.775118 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.774215    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 07 11:24:29.775118 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.774289    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 11:24:29.781087 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.781064    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"]
Sep 07 11:24:29.806088 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.806062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-hub\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.806202 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.806092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.806202 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.806134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cflm\" (UniqueName: \"kubernetes.io/projected/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-kube-api-access-4cflm\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.806202 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.806162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.806202 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.806194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.806363 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.806211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-ca\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.907024 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.906995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-hub\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.907170 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.907031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.907170 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.907075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cflm\" (UniqueName: \"kubernetes.io/projected/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-kube-api-access-4cflm\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.907170 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.907105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.907170 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.907141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.907392 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.907261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-ca\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.907975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.907949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.909629 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.909593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-ca\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.909629 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.909611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-hub\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.909901 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.909883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.910089 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.910071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:29.914486 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:29.914462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cflm\" (UniqueName: \"kubernetes.io/projected/ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0-kube-api-access-4cflm\") pod \"cluster-proxy-proxy-agent-5cc488897d-srwq6\" (UID: \"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:30.092581 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:30.092497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"
Sep 07 11:24:30.367887 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:30.367816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6"]
Sep 07 11:24:30.370630 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:24:30.370600    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce7f8fc4_1684_46b3_ac73_fe8bc60ab0e0.slice/crio-75e4e32e0ad41f2c67397621fa52820fa6a38e15ecd2ce0c56d94d68293c36e3 WatchSource:0}: Error finding container 75e4e32e0ad41f2c67397621fa52820fa6a38e15ecd2ce0c56d94d68293c36e3: Status 404 returned error can't find the container with id 75e4e32e0ad41f2c67397621fa52820fa6a38e15ecd2ce0c56d94d68293c36e3
Sep 07 11:24:31.330523 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:31.330458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6" event={"ID":"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0","Type":"ContainerStarted","Data":"75e4e32e0ad41f2c67397621fa52820fa6a38e15ecd2ce0c56d94d68293c36e3"}
Sep 07 11:24:34.342693 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:34.342607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6" event={"ID":"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0","Type":"ContainerStarted","Data":"f65cb613abe6956b9257934742ba46d9d5ef4397f25a68c7a2a4a5eaada58a14"}
Sep 07 11:24:34.342693 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:34.342653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6" event={"ID":"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0","Type":"ContainerStarted","Data":"f10490f15f1a0fd33644dfa157758f921929f639665bea5cd4ebe97ef6037698"}
Sep 07 11:24:34.342693 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:34.342667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6" event={"ID":"ce7f8fc4-1684-46b3-ac73-fe8bc60ab0e0","Type":"ContainerStarted","Data":"0795f8c1ece3fa855ac2c7381babac745ec88fe3a200bb0df7010039da0e0736"}
Sep 07 11:24:34.359951 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:34.359906    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc488897d-srwq6" podStartSLOduration=1.730899661 podStartE2EDuration="5.35989176s" totalImagesPullingTime="3.628992099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:34.35963046 +0000 UTC m=+124.065411162" watchObservedRunningTime="2026-09-07 11:24:34.35989176 +0000 UTC m=+124.065672451"
Sep 07 11:24:40.585670 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:40.585626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:24:40.587870 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:40.587850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e5bbff-0cc8-45cb-91b8-b106b0a5a978-metrics-certs\") pod \"network-metrics-daemon-lmtxp\" (UID: \"85e5bbff-0cc8-45cb-91b8-b106b0a5a978\") " pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:24:40.746384 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:40.746351    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qlhnv\""
Sep 07 11:24:40.754056 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:40.754038    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmtxp"
Sep 07 11:24:41.022234 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:41.022211    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lmtxp"]
Sep 07 11:24:41.024358 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:24:41.024329    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e5bbff_0cc8_45cb_91b8_b106b0a5a978.slice/crio-91e47433a1a782c21a723c5e5819f88b39fd88f856480690b4b14efdf6894baf WatchSource:0}: Error finding container 91e47433a1a782c21a723c5e5819f88b39fd88f856480690b4b14efdf6894baf: Status 404 returned error can't find the container with id 91e47433a1a782c21a723c5e5819f88b39fd88f856480690b4b14efdf6894baf
Sep 07 11:24:41.364173 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:41.364089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lmtxp" event={"ID":"85e5bbff-0cc8-45cb-91b8-b106b0a5a978","Type":"ContainerStarted","Data":"91e47433a1a782c21a723c5e5819f88b39fd88f856480690b4b14efdf6894baf"}
Sep 07 11:24:42.369554 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:42.369515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lmtxp" event={"ID":"85e5bbff-0cc8-45cb-91b8-b106b0a5a978","Type":"ContainerStarted","Data":"dc9f41ff6852470a539fbf2baaedbce70a47178bd3cf55c3258d7669aa4571b2"}
Sep 07 11:24:42.369925 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:42.369621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lmtxp" event={"ID":"85e5bbff-0cc8-45cb-91b8-b106b0a5a978","Type":"ContainerStarted","Data":"cdad01535339f24dd400ad027222138a370116babc22362432e2adc23eb3f472"}
Sep 07 11:24:42.384255 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:42.384208    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lmtxp" podStartSLOduration=130.218127221 podStartE2EDuration="2m11.384190251s" totalImagesPullingTime="1.16606303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:42.384104305 +0000 UTC m=+132.089885009" watchObservedRunningTime="2026-09-07 11:24:42.384190251 +0000 UTC m=+132.089970942"
Sep 07 11:24:51.955408 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:51.955372    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6968776c5f-zfzqj"]
Sep 07 11:24:51.955776 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:51.955577    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6968776c5f-zfzqj" podUID="481046bf-a011-484e-8e20-18b2d7edc210"
Sep 07 11:24:52.401576 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.401547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:24:52.407511 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.407488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:24:52.475706 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.475682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s49v\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-kube-api-access-9s49v\") pod \"481046bf-a011-484e-8e20-18b2d7edc210\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") "
Sep 07 11:24:52.475706 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.475713    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-trusted-ca\") pod \"481046bf-a011-484e-8e20-18b2d7edc210\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") "
Sep 07 11:24:52.475947 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.475734    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/481046bf-a011-484e-8e20-18b2d7edc210-ca-trust-extracted\") pod \"481046bf-a011-484e-8e20-18b2d7edc210\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") "
Sep 07 11:24:52.475947 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.475758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-registry-certificates\") pod \"481046bf-a011-484e-8e20-18b2d7edc210\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") "
Sep 07 11:24:52.475947 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.475819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-image-registry-private-configuration\") pod \"481046bf-a011-484e-8e20-18b2d7edc210\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") "
Sep 07 11:24:52.475947 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.475853    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-installation-pull-secrets\") pod \"481046bf-a011-484e-8e20-18b2d7edc210\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") "
Sep 07 11:24:52.475947 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.475901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-bound-sa-token\") pod \"481046bf-a011-484e-8e20-18b2d7edc210\" (UID: \"481046bf-a011-484e-8e20-18b2d7edc210\") "
Sep 07 11:24:52.476190 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.476113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481046bf-a011-484e-8e20-18b2d7edc210-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "481046bf-a011-484e-8e20-18b2d7edc210" (UID: "481046bf-a011-484e-8e20-18b2d7edc210"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:24:52.476308 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.476277    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "481046bf-a011-484e-8e20-18b2d7edc210" (UID: "481046bf-a011-484e-8e20-18b2d7edc210"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:24:52.476442 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.476285    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "481046bf-a011-484e-8e20-18b2d7edc210" (UID: "481046bf-a011-484e-8e20-18b2d7edc210"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:24:52.478093 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.478065    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "481046bf-a011-484e-8e20-18b2d7edc210" (UID: "481046bf-a011-484e-8e20-18b2d7edc210"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:24:52.478174 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.478105    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "481046bf-a011-484e-8e20-18b2d7edc210" (UID: "481046bf-a011-484e-8e20-18b2d7edc210"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:24:52.478174 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.478160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "481046bf-a011-484e-8e20-18b2d7edc210" (UID: "481046bf-a011-484e-8e20-18b2d7edc210"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:24:52.478246 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.478163    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-kube-api-access-9s49v" (OuterVolumeSpecName: "kube-api-access-9s49v") pod "481046bf-a011-484e-8e20-18b2d7edc210" (UID: "481046bf-a011-484e-8e20-18b2d7edc210"). InnerVolumeSpecName "kube-api-access-9s49v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:24:52.577102 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.577070    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s49v\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-kube-api-access-9s49v\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:24:52.577102 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.577098    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-trusted-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:24:52.577102 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.577108    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/481046bf-a011-484e-8e20-18b2d7edc210-ca-trust-extracted\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:24:52.577308 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.577116    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/481046bf-a011-484e-8e20-18b2d7edc210-registry-certificates\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:24:52.577308 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.577125    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-image-registry-private-configuration\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:24:52.577308 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.577136    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/481046bf-a011-484e-8e20-18b2d7edc210-installation-pull-secrets\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:24:52.577308 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.577145    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-bound-sa-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:24:52.972561 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.972532    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fh8c5"]
Sep 07 11:24:52.978319 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.978295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:52.980734 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.980710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 11:24:52.981010 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.980994    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 11:24:52.981200 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.981186    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 11:24:52.981281 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.981263    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 11:24:52.981970 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.981951    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wb972\""
Sep 07 11:24:52.982052 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.981984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 11:24:52.982052 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:52.981997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 11:24:53.081702 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.081702 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-textfile\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.081937 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-tls\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.081937 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aadab419-cf81-41df-8c04-d0a8c224205e-metrics-client-ca\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.081937 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-root\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.081937 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqk4\" (UniqueName: \"kubernetes.io/projected/aadab419-cf81-41df-8c04-d0a8c224205e-kube-api-access-drqk4\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.081937 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.081937 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-sys\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.082124 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.081957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-wtmp\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.182968 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.182925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.183134 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-textfile\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.183351 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.183351 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-textfile\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.183531 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-tls\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.183531 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aadab419-cf81-41df-8c04-d0a8c224205e-metrics-client-ca\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.183531 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-root\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.183936 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aadab419-cf81-41df-8c04-d0a8c224205e-metrics-client-ca\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.183936 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqk4\" (UniqueName: \"kubernetes.io/projected/aadab419-cf81-41df-8c04-d0a8c224205e-kube-api-access-drqk4\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.184108 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-root\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.184108 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:53.183543    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 07 11:24:53.184108 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.183974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.184108 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:53.184008    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-tls podName:aadab419-cf81-41df-8c04-d0a8c224205e nodeName:}" failed. No retries permitted until 2026-09-07 11:24:53.683988296 +0000 UTC m=+143.389768968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-tls") pod "node-exporter-fh8c5" (UID: "aadab419-cf81-41df-8c04-d0a8c224205e") : secret "node-exporter-tls" not found
Sep 07 11:24:53.184108 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.184047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-sys\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.184108 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.184107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-wtmp\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.184397 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.184245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-wtmp\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.184397 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.184289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aadab419-cf81-41df-8c04-d0a8c224205e-sys\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.186336 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.186311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.191997 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.191972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqk4\" (UniqueName: \"kubernetes.io/projected/aadab419-cf81-41df-8c04-d0a8c224205e-kube-api-access-drqk4\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.404306 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.404274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6968776c5f-zfzqj"
Sep 07 11:24:53.435457 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.435424    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6968776c5f-zfzqj"]
Sep 07 11:24:53.439317 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.439291    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6968776c5f-zfzqj"]
Sep 07 11:24:53.486508 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.486478    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/481046bf-a011-484e-8e20-18b2d7edc210-registry-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:24:53.688045 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.687963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-tls\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.690333 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.690310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aadab419-cf81-41df-8c04-d0a8c224205e-node-exporter-tls\") pod \"node-exporter-fh8c5\" (UID: \"aadab419-cf81-41df-8c04-d0a8c224205e\") " pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.887998 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:53.887971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fh8c5"
Sep 07 11:24:53.905539 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:24:53.905509    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadab419_cf81_41df_8c04_d0a8c224205e.slice/crio-90cb0159fb55c89d1b52fb1b88243f0ba760c087ae45e18ab84d356d2a9451ca WatchSource:0}: Error finding container 90cb0159fb55c89d1b52fb1b88243f0ba760c087ae45e18ab84d356d2a9451ca: Status 404 returned error can't find the container with id 90cb0159fb55c89d1b52fb1b88243f0ba760c087ae45e18ab84d356d2a9451ca
Sep 07 11:24:54.033203 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.033169    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 11:24:54.037015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.036993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.039509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.039485    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mrk7x\""
Sep 07 11:24:54.039619 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.039492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 07 11:24:54.039762 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.039746    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 07 11:24:54.039836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.039766    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 07 11:24:54.039836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.039748    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 07 11:24:54.040023 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.040005    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 07 11:24:54.040158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.040079    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 07 11:24:54.040158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.040095    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 07 11:24:54.040158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.040088    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 07 11:24:54.040158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.040108    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 07 11:24:54.050588 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.050567    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 11:24:54.090827 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.090783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-web-config\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.090827 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.090830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.090972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.090850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.090972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.090905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-config-out\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.090972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.090929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnt84\" (UniqueName: \"kubernetes.io/projected/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-kube-api-access-mnt84\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.090972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.090968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.091091 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.090989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.091091 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.091009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.091091 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.091062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.091188 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.091092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.091188 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.091113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.091188 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.091132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.091188 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.091165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.191748 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.191715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.191922 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.191755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.191922 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.191781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.191922 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.191840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.191922 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.191867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192102 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192160 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192160 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192260 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-web-config\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192260 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192260 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192410 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-config-out\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192410 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnt84\" (UniqueName: \"kubernetes.io/projected/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-kube-api-access-mnt84\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192504 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192780 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.192960 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.192827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.195435 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.195410    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.195533 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.195476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.195667 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.195651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-config-out\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.195729 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.195652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.195729 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.195709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.195862 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.195784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.195960 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.195939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.195997 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.195962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.196737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.196715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-web-config\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.209262 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.209242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnt84\" (UniqueName: \"kubernetes.io/projected/ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37-kube-api-access-mnt84\") pod \"alertmanager-main-0\" (UID: \"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.346647 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.346564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 11:24:54.412916 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.412879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh8c5" event={"ID":"aadab419-cf81-41df-8c04-d0a8c224205e","Type":"ContainerStarted","Data":"90cb0159fb55c89d1b52fb1b88243f0ba760c087ae45e18ab84d356d2a9451ca"}
Sep 07 11:24:54.627639 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.627565    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 11:24:54.631075 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:24:54.631046    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2efefd_a8a5_4afe_ae0e_40a7f0fc3a37.slice/crio-df68fc2bbf219e7a1c4b30bc9c01217519fb3493a09a47e0215ae945990ff935 WatchSource:0}: Error finding container df68fc2bbf219e7a1c4b30bc9c01217519fb3493a09a47e0215ae945990ff935: Status 404 returned error can't find the container with id df68fc2bbf219e7a1c4b30bc9c01217519fb3493a09a47e0215ae945990ff935
Sep 07 11:24:54.938978 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:54.938899    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481046bf-a011-484e-8e20-18b2d7edc210" path="/var/lib/kubelet/pods/481046bf-a011-484e-8e20-18b2d7edc210/volumes"
Sep 07 11:24:55.058338 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.058304    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d74f69f95-txmmm"]
Sep 07 11:24:55.061828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.061781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.064060 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.064034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 07 11:24:55.064186 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.064125    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4qatsjk7gn8hd\""
Sep 07 11:24:55.064347 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.064330    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 07 11:24:55.064347 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.064331    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 07 11:24:55.065045 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.065024    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 07 11:24:55.065111 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.065080    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 07 11:24:55.065179 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.065161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cbp8v\""
Sep 07 11:24:55.071042 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.071021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d74f69f95-txmmm"]
Sep 07 11:24:55.201814 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.201727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.201814 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.201764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8n4\" (UniqueName: \"kubernetes.io/projected/61a57663-07ba-4c20-a30f-7b0bae9f52e5-kube-api-access-ff8n4\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.201814 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.201805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.202024 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.201831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a57663-07ba-4c20-a30f-7b0bae9f52e5-metrics-client-ca\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.202024 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.201904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-grpc-tls\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.202024 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.202007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.202136 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.202059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-tls\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.202136 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.202088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.302808 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.302760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.302962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.302834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-tls\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.302962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.302866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.302962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.302925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.302962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.302951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8n4\" (UniqueName: \"kubernetes.io/projected/61a57663-07ba-4c20-a30f-7b0bae9f52e5-kube-api-access-ff8n4\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.303137 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.302980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.303137 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.303006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a57663-07ba-4c20-a30f-7b0bae9f52e5-metrics-client-ca\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.303137 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.303031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-grpc-tls\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.303990 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.303955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a57663-07ba-4c20-a30f-7b0bae9f52e5-metrics-client-ca\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.305674 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.305650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-tls\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.305895 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.305868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.306010 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.305986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-grpc-tls\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.306435 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.306413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.306585 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.306413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.306688 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.306584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/61a57663-07ba-4c20-a30f-7b0bae9f52e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.313488 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.313465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8n4\" (UniqueName: \"kubernetes.io/projected/61a57663-07ba-4c20-a30f-7b0bae9f52e5-kube-api-access-ff8n4\") pod \"thanos-querier-5d74f69f95-txmmm\" (UID: \"61a57663-07ba-4c20-a30f-7b0bae9f52e5\") " pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.370631 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.370596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:55.420908 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.420783    2581 generic.go:356] "Generic (PLEG): container finished" podID="aadab419-cf81-41df-8c04-d0a8c224205e" containerID="152100b90045a31379d958acb7c985f19bc3efd13d7df36fedb841010e3fcdfa" exitCode=0
Sep 07 11:24:55.421287 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.421244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh8c5" event={"ID":"aadab419-cf81-41df-8c04-d0a8c224205e","Type":"ContainerDied","Data":"152100b90045a31379d958acb7c985f19bc3efd13d7df36fedb841010e3fcdfa"}
Sep 07 11:24:55.424066 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.424033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37","Type":"ContainerStarted","Data":"df68fc2bbf219e7a1c4b30bc9c01217519fb3493a09a47e0215ae945990ff935"}
Sep 07 11:24:55.644400 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:55.644364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d74f69f95-txmmm"]
Sep 07 11:24:55.776999 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:24:55.776962    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a57663_07ba_4c20_a30f_7b0bae9f52e5.slice/crio-745c0fb8395e47e21f7cf7613b6241cb89d42dcc1a26ed9d6f101b17d67013cb WatchSource:0}: Error finding container 745c0fb8395e47e21f7cf7613b6241cb89d42dcc1a26ed9d6f101b17d67013cb: Status 404 returned error can't find the container with id 745c0fb8395e47e21f7cf7613b6241cb89d42dcc1a26ed9d6f101b17d67013cb
Sep 07 11:24:56.429191 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:56.429154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh8c5" event={"ID":"aadab419-cf81-41df-8c04-d0a8c224205e","Type":"ContainerStarted","Data":"38ff0480b018666c49e41844286c99d261daef93af4faa324745ce1b0a47dc65"}
Sep 07 11:24:56.429740 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:56.429213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh8c5" event={"ID":"aadab419-cf81-41df-8c04-d0a8c224205e","Type":"ContainerStarted","Data":"b61db36d0e350fac77cfeca9f0b3a72f8584750eb9417255271a681b1bfa825d"}
Sep 07 11:24:56.430388 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:56.430362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm" event={"ID":"61a57663-07ba-4c20-a30f-7b0bae9f52e5","Type":"ContainerStarted","Data":"745c0fb8395e47e21f7cf7613b6241cb89d42dcc1a26ed9d6f101b17d67013cb"}
Sep 07 11:24:56.431856 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:56.431830    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37" containerID="babebb04eed6489f8a61ee92efd3a8754a313ebc4b7012f9c6736bfa4d8fbc3f" exitCode=0
Sep 07 11:24:56.431940 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:56.431903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37","Type":"ContainerDied","Data":"babebb04eed6489f8a61ee92efd3a8754a313ebc4b7012f9c6736bfa4d8fbc3f"}
Sep 07 11:24:56.448714 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:56.448676    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fh8c5" podStartSLOduration=3.204514861 podStartE2EDuration="4.44866262s" totalImagesPullingTime="1.183279501s" totalInitContainerRuntime="60.868258ms" isStatefulPod=true podCreationTimestamp="2026-09-07 11:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:56.448427777 +0000 UTC m=+146.154208466" watchObservedRunningTime="2026-09-07 11:24:56.44866262 +0000 UTC m=+146.154443310"
Sep 07 11:24:57.768765 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.768731    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"]
Sep 07 11:24:57.771626 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.771609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:57.773886 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.773863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 07 11:24:57.773991 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.773938    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2krkl\""
Sep 07 11:24:57.780251 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.780229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"]
Sep 07 11:24:57.822307 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.822282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-spjrj\" (UID: \"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:57.822454 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.822323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddm8\" (UniqueName: \"kubernetes.io/projected/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-kube-api-access-5ddm8\") pod \"monitoring-plugin-85bbbbbf4-spjrj\" (UID: \"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:57.923784 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.923755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-spjrj\" (UID: \"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:57.923892 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.923833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddm8\" (UniqueName: \"kubernetes.io/projected/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-kube-api-access-5ddm8\") pod \"monitoring-plugin-85bbbbbf4-spjrj\" (UID: \"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:57.923940 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:57.923899    2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 07 11:24:57.923993 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:24:57.923972    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-monitoring-plugin-cert podName:5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83 nodeName:}" failed. No retries permitted until 2026-09-07 11:24:58.423950547 +0000 UTC m=+148.129731215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-spjrj" (UID: "5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83") : secret "monitoring-plugin-cert" not found
Sep 07 11:24:57.932839 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:57.932814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddm8\" (UniqueName: \"kubernetes.io/projected/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-kube-api-access-5ddm8\") pod \"monitoring-plugin-85bbbbbf4-spjrj\" (UID: \"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:58.428975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:58.428950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-spjrj\" (UID: \"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:58.430936 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:58.430915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-spjrj\" (UID: \"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:58.442437 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:58.442404    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm" event={"ID":"61a57663-07ba-4c20-a30f-7b0bae9f52e5","Type":"ContainerStarted","Data":"edccde53129aacd931fc5d418299e6b5348f1f28c3180b47bcc65c87d917f8f4"}
Sep 07 11:24:58.442539 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:58.442488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm" event={"ID":"61a57663-07ba-4c20-a30f-7b0bae9f52e5","Type":"ContainerStarted","Data":"c925ddda2922de2691badb6fcaf736636cd01420c4f0f83eee634590b1063276"}
Sep 07 11:24:58.442539 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:58.442534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm" event={"ID":"61a57663-07ba-4c20-a30f-7b0bae9f52e5","Type":"ContainerStarted","Data":"eef18776da6fe3034894f3fa456a9e5005f4551f600c37c8bff58b9b0ad8f836"}
Sep 07 11:24:58.445079 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:58.445058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37","Type":"ContainerStarted","Data":"76392bc01ab277a9be54230d06dcd4968b640d346e62dae5c54a25fc6e808fe0"}
Sep 07 11:24:58.681345 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:58.681312    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:24:58.949254 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:58.949172    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"]
Sep 07 11:24:58.952577 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:24:58.952545    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1f75c4_57e6_4e6b_98fa_9640c4a6ce83.slice/crio-0c4d3bcd8dc48c6ebdef5e73c88a194b29e4f91497029dcb8611e419c71e8de1 WatchSource:0}: Error finding container 0c4d3bcd8dc48c6ebdef5e73c88a194b29e4f91497029dcb8611e419c71e8de1: Status 404 returned error can't find the container with id 0c4d3bcd8dc48c6ebdef5e73c88a194b29e4f91497029dcb8611e419c71e8de1
Sep 07 11:24:59.450278 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.450238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj" event={"ID":"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83","Type":"ContainerStarted","Data":"0c4d3bcd8dc48c6ebdef5e73c88a194b29e4f91497029dcb8611e419c71e8de1"}
Sep 07 11:24:59.453271 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.453193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm" event={"ID":"61a57663-07ba-4c20-a30f-7b0bae9f52e5","Type":"ContainerStarted","Data":"519d947c73f2cfb7b4cd2a3f4517437c99cae83bf390c142037c3ca0f737e85c"}
Sep 07 11:24:59.453271 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.453253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm" event={"ID":"61a57663-07ba-4c20-a30f-7b0bae9f52e5","Type":"ContainerStarted","Data":"18828464be8e84d0b29dd82732575b7c40597b07b45248d4fa0deaf878a59e3c"}
Sep 07 11:24:59.453444 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.453287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm" event={"ID":"61a57663-07ba-4c20-a30f-7b0bae9f52e5","Type":"ContainerStarted","Data":"b8e0286fb37ce14cfeef7377d3768db77924e05552c385a008ef08873b4fdf77"}
Sep 07 11:24:59.453444 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.453324    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:24:59.456350 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.456329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37","Type":"ContainerStarted","Data":"ae65a07292f81d02f93903a8ff1d96ad4f5a73f44048decb456a22e29bd1f490"}
Sep 07 11:24:59.456434 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.456381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37","Type":"ContainerStarted","Data":"00815b208f404a1476d6a04544b38cf5b622de868426b8a242da3707a0375f35"}
Sep 07 11:24:59.456434 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.456406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37","Type":"ContainerStarted","Data":"a8cd75c4048357c88dbf76760fc9fedf747bf2eddf24800e864bd9be8b1a4b64"}
Sep 07 11:24:59.456502 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.456437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37","Type":"ContainerStarted","Data":"24f10966e25ba6713309ddacce1befa526de9938974d44adb626894c878dc5b8"}
Sep 07 11:24:59.456502 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.456468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37","Type":"ContainerStarted","Data":"e55960ea66a9eddc7c94b91a3062d1d4b9de411fd9bcc7355794beb146560401"}
Sep 07 11:24:59.476257 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.476210    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm" podStartSLOduration=1.262473799 podStartE2EDuration="4.476195776s" totalImagesPullingTime="3.213721977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:59.474431212 +0000 UTC m=+149.180211904" watchObservedRunningTime="2026-09-07 11:24:59.476195776 +0000 UTC m=+149.181976465"
Sep 07 11:24:59.497884 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:24:59.497836    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.80412338 podStartE2EDuration="5.497821549s" totalImagesPullingTime="3.676770142s" totalInitContainerRuntime="16.928027ms" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:24:59.496669683 +0000 UTC m=+149.202450375" watchObservedRunningTime="2026-09-07 11:24:59.497821549 +0000 UTC m=+149.203602238"
Sep 07 11:25:00.461441 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:00.461399    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj" event={"ID":"5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83","Type":"ContainerStarted","Data":"fbb259c12d8f4482aacd9d75f292dc71c13a7fc997791d744777b22fc2136b7e"}
Sep 07 11:25:00.462165 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:00.462145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:25:00.466476 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:00.466454    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj"
Sep 07 11:25:00.476561 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:00.476523    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-spjrj" podStartSLOduration=2.095858055 podStartE2EDuration="3.476510831s" totalImagesPullingTime="1.380652776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:25:00.475990529 +0000 UTC m=+150.181771219" watchObservedRunningTime="2026-09-07 11:25:00.476510831 +0000 UTC m=+150.182291520"
Sep 07 11:25:05.467155 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:05.467125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d74f69f95-txmmm"
Sep 07 11:25:08.631018 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:25:08.630957    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-h9qtv" podUID="8aecb463-b248-47aa-8ac7-0e15b220c491"
Sep 07 11:25:09.494652 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:09.494623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:25:13.648259 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:13.648211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:25:13.650550 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:13.650524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aecb463-b248-47aa-8ac7-0e15b220c491-metrics-tls\") pod \"dns-default-h9qtv\" (UID: \"8aecb463-b248-47aa-8ac7-0e15b220c491\") " pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:25:13.698711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:13.698686    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ph89j\""
Sep 07 11:25:13.706785 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:13.706768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:25:13.977845 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:13.977786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h9qtv"]
Sep 07 11:25:13.980432 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:25:13.980391    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aecb463_b248_47aa_8ac7_0e15b220c491.slice/crio-9132f526aae4f58ed569cec02c725a9f4a6370f31375ba2ba7d4f1dac384f8fa WatchSource:0}: Error finding container 9132f526aae4f58ed569cec02c725a9f4a6370f31375ba2ba7d4f1dac384f8fa: Status 404 returned error can't find the container with id 9132f526aae4f58ed569cec02c725a9f4a6370f31375ba2ba7d4f1dac384f8fa
Sep 07 11:25:14.513633 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:14.513598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9qtv" event={"ID":"8aecb463-b248-47aa-8ac7-0e15b220c491","Type":"ContainerStarted","Data":"9132f526aae4f58ed569cec02c725a9f4a6370f31375ba2ba7d4f1dac384f8fa"}
Sep 07 11:25:16.522179 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:16.522142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9qtv" event={"ID":"8aecb463-b248-47aa-8ac7-0e15b220c491","Type":"ContainerStarted","Data":"b32378cafbf989e0ab8fbfc113bd97bd864e204ab20f965fd0bf8fb3e496d859"}
Sep 07 11:25:16.522583 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:16.522205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9qtv" event={"ID":"8aecb463-b248-47aa-8ac7-0e15b220c491","Type":"ContainerStarted","Data":"3240043272b429dfd8d0550d0a0f7d8e4dbbad06f1a5a3ec0621bd53e1718894"}
Sep 07 11:25:16.522583 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:16.522266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:25:16.538621 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:16.538578    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h9qtv" podStartSLOduration=129.837037443 podStartE2EDuration="2m11.53856405s" totalImagesPullingTime="1.701526607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:25:16.537769456 +0000 UTC m=+166.243550146" watchObservedRunningTime="2026-09-07 11:25:16.53856405 +0000 UTC m=+166.244344786"
Sep 07 11:25:26.528317 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:25:26.528283    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h9qtv"
Sep 07 11:27:30.723678 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:27:30.723647    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 11:28:02.083260 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.081313    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"]
Sep 07 11:28:02.085320 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.085288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"
Sep 07 11:28:02.087534 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.087507    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:28:02.087641 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.087514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zhxk9\""
Sep 07 11:28:02.087641 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.087559    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 07 11:28:02.093782 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.093760    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"]
Sep 07 11:28:02.109709 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.109685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d834b6eb-adee-4406-8849-26ff930b9169-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sf7zx\" (UID: \"d834b6eb-adee-4406-8849-26ff930b9169\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"
Sep 07 11:28:02.109818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.109712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dct5k\" (UniqueName: \"kubernetes.io/projected/d834b6eb-adee-4406-8849-26ff930b9169-kube-api-access-dct5k\") pod \"cert-manager-operator-controller-manager-55c444bf86-sf7zx\" (UID: \"d834b6eb-adee-4406-8849-26ff930b9169\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"
Sep 07 11:28:02.210398 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.210360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d834b6eb-adee-4406-8849-26ff930b9169-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sf7zx\" (UID: \"d834b6eb-adee-4406-8849-26ff930b9169\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"
Sep 07 11:28:02.210555 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.210505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dct5k\" (UniqueName: \"kubernetes.io/projected/d834b6eb-adee-4406-8849-26ff930b9169-kube-api-access-dct5k\") pod \"cert-manager-operator-controller-manager-55c444bf86-sf7zx\" (UID: \"d834b6eb-adee-4406-8849-26ff930b9169\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"
Sep 07 11:28:02.210756 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.210737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d834b6eb-adee-4406-8849-26ff930b9169-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sf7zx\" (UID: \"d834b6eb-adee-4406-8849-26ff930b9169\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"
Sep 07 11:28:02.219296 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.219271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dct5k\" (UniqueName: \"kubernetes.io/projected/d834b6eb-adee-4406-8849-26ff930b9169-kube-api-access-dct5k\") pod \"cert-manager-operator-controller-manager-55c444bf86-sf7zx\" (UID: \"d834b6eb-adee-4406-8849-26ff930b9169\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"
Sep 07 11:28:02.394454 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.394341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"
Sep 07 11:28:02.674749 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.674726    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx"]
Sep 07 11:28:02.677494 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:28:02.677466    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd834b6eb_adee_4406_8849_26ff930b9169.slice/crio-7438ab0e31c3272df838f861ef6ae77c734355512971f51790c2a59c18b1cb02 WatchSource:0}: Error finding container 7438ab0e31c3272df838f861ef6ae77c734355512971f51790c2a59c18b1cb02: Status 404 returned error can't find the container with id 7438ab0e31c3272df838f861ef6ae77c734355512971f51790c2a59c18b1cb02
Sep 07 11:28:02.680198 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:02.680181    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:28:03.110164 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:03.110129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx" event={"ID":"d834b6eb-adee-4406-8849-26ff930b9169","Type":"ContainerStarted","Data":"7438ab0e31c3272df838f861ef6ae77c734355512971f51790c2a59c18b1cb02"}
Sep 07 11:28:06.123707 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:06.123669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx" event={"ID":"d834b6eb-adee-4406-8849-26ff930b9169","Type":"ContainerStarted","Data":"88d3922a8a43ba01aeab6df0c9bf5da46161cf2f892a3c97428d57dfbc2d09ee"}
Sep 07 11:28:06.149916 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:06.149868    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sf7zx" podStartSLOduration=1.315078825 podStartE2EDuration="4.149852875s" totalImagesPullingTime="2.83477405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:28:06.148111187 +0000 UTC m=+335.853891879" watchObservedRunningTime="2026-09-07 11:28:06.149852875 +0000 UTC m=+335.855633565"
Sep 07 11:28:20.176264 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.176227    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"]
Sep 07 11:28:20.179513 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.179497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"
Sep 07 11:28:20.181853 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.181824    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 11:28:20.181964 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.181835    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 11:28:20.182568 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.182545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jhkqs\""
Sep 07 11:28:20.187732 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.187709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"]
Sep 07 11:28:20.237768 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.237731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6kst\" (UniqueName: \"kubernetes.io/projected/d2d38264-0d09-4638-8779-917c2600ac79-kube-api-access-q6kst\") pod \"openshift-lws-operator-fd8ccff4c-jwkst\" (UID: \"d2d38264-0d09-4638-8779-917c2600ac79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"
Sep 07 11:28:20.237933 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.237784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2d38264-0d09-4638-8779-917c2600ac79-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jwkst\" (UID: \"d2d38264-0d09-4638-8779-917c2600ac79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"
Sep 07 11:28:20.339011 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.338982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6kst\" (UniqueName: \"kubernetes.io/projected/d2d38264-0d09-4638-8779-917c2600ac79-kube-api-access-q6kst\") pod \"openshift-lws-operator-fd8ccff4c-jwkst\" (UID: \"d2d38264-0d09-4638-8779-917c2600ac79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"
Sep 07 11:28:20.339129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.339021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2d38264-0d09-4638-8779-917c2600ac79-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jwkst\" (UID: \"d2d38264-0d09-4638-8779-917c2600ac79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"
Sep 07 11:28:20.339361 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.339346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2d38264-0d09-4638-8779-917c2600ac79-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jwkst\" (UID: \"d2d38264-0d09-4638-8779-917c2600ac79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"
Sep 07 11:28:20.346808 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.346773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6kst\" (UniqueName: \"kubernetes.io/projected/d2d38264-0d09-4638-8779-917c2600ac79-kube-api-access-q6kst\") pod \"openshift-lws-operator-fd8ccff4c-jwkst\" (UID: \"d2d38264-0d09-4638-8779-917c2600ac79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"
Sep 07 11:28:20.488722 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.488697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"
Sep 07 11:28:20.756257 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:20.756211    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst"]
Sep 07 11:28:20.758704 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:28:20.758670    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2d38264_0d09_4638_8779_917c2600ac79.slice/crio-b6c305799e8818fe75f77be86d661e483e96524be4e1cf8af99efada6b375dd0 WatchSource:0}: Error finding container b6c305799e8818fe75f77be86d661e483e96524be4e1cf8af99efada6b375dd0: Status 404 returned error can't find the container with id b6c305799e8818fe75f77be86d661e483e96524be4e1cf8af99efada6b375dd0
Sep 07 11:28:21.179438 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:21.179359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst" event={"ID":"d2d38264-0d09-4638-8779-917c2600ac79","Type":"ContainerStarted","Data":"b6c305799e8818fe75f77be86d661e483e96524be4e1cf8af99efada6b375dd0"}
Sep 07 11:28:24.193706 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:24.193663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst" event={"ID":"d2d38264-0d09-4638-8779-917c2600ac79","Type":"ContainerStarted","Data":"6b38ec138859231d0e5fc644f7694f8b5cc44d8d83683fe0006c867b3cd6f698"}
Sep 07 11:28:24.214954 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:24.214911    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jwkst" podStartSLOduration=1.6340974099999999 podStartE2EDuration="4.214896622s" totalImagesPullingTime="2.580799212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:28:24.212613517 +0000 UTC m=+353.918394207" watchObservedRunningTime="2026-09-07 11:28:24.214896622 +0000 UTC m=+353.920677312"
Sep 07 11:28:45.740297 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.740262    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"]
Sep 07 11:28:45.745916 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.745893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.749567 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.749536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 11:28:45.749835 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.749807    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 11:28:45.749835 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.749824    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 11:28:45.749957 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.749864    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g8bwf\""
Sep 07 11:28:45.752366 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.752349    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 11:28:45.765703 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.765679    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"]
Sep 07 11:28:45.835628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.835598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlwcf\" (UniqueName: \"kubernetes.io/projected/a76fc137-c23c-4d98-9b77-a769f24eafde-kube-api-access-zlwcf\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.835628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.835629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a76fc137-c23c-4d98-9b77-a769f24eafde-manifests\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.835834 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.835676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a76fc137-c23c-4d98-9b77-a769f24eafde-webhook-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.835834 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.835700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a76fc137-c23c-4d98-9b77-a769f24eafde-tmp\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.835834 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.835721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a76fc137-c23c-4d98-9b77-a769f24eafde-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.936239 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.936200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a76fc137-c23c-4d98-9b77-a769f24eafde-webhook-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.936239 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.936247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a76fc137-c23c-4d98-9b77-a769f24eafde-tmp\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.936475 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.936278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a76fc137-c23c-4d98-9b77-a769f24eafde-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.936475 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.936312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlwcf\" (UniqueName: \"kubernetes.io/projected/a76fc137-c23c-4d98-9b77-a769f24eafde-kube-api-access-zlwcf\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.936475 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.936337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a76fc137-c23c-4d98-9b77-a769f24eafde-manifests\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.936782 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.936761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a76fc137-c23c-4d98-9b77-a769f24eafde-manifests\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.938630 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.938608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a76fc137-c23c-4d98-9b77-a769f24eafde-tmp\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.938999 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.938977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a76fc137-c23c-4d98-9b77-a769f24eafde-webhook-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.939042 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.938982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a76fc137-c23c-4d98-9b77-a769f24eafde-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:45.947049 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:45.947027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlwcf\" (UniqueName: \"kubernetes.io/projected/a76fc137-c23c-4d98-9b77-a769f24eafde-kube-api-access-zlwcf\") pod \"opendatahub-operator-controller-manager-f479d7474-bz4hv\" (UID: \"a76fc137-c23c-4d98-9b77-a769f24eafde\") " pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:46.055658 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:46.055579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:46.331500 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:46.331471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"]
Sep 07 11:28:46.334143 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:28:46.334118    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76fc137_c23c_4d98_9b77_a769f24eafde.slice/crio-38d6dbed49393d803500ef8ddd1ebc912d6de90dffe90fd5cf36b4acc365c05c WatchSource:0}: Error finding container 38d6dbed49393d803500ef8ddd1ebc912d6de90dffe90fd5cf36b4acc365c05c: Status 404 returned error can't find the container with id 38d6dbed49393d803500ef8ddd1ebc912d6de90dffe90fd5cf36b4acc365c05c
Sep 07 11:28:47.285715 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:47.285638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv" event={"ID":"a76fc137-c23c-4d98-9b77-a769f24eafde","Type":"ContainerStarted","Data":"38d6dbed49393d803500ef8ddd1ebc912d6de90dffe90fd5cf36b4acc365c05c"}
Sep 07 11:28:50.301024 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:50.300987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv" event={"ID":"a76fc137-c23c-4d98-9b77-a769f24eafde","Type":"ContainerStarted","Data":"b583cba332efa9a9511da12e08a78f8ccf67a11d686f3d6e3e888d63ae3d54fd"}
Sep 07 11:28:51.306350 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:51.306318    2581 generic.go:356] "Generic (PLEG): container finished" podID="a76fc137-c23c-4d98-9b77-a769f24eafde" containerID="b583cba332efa9a9511da12e08a78f8ccf67a11d686f3d6e3e888d63ae3d54fd" exitCode=0
Sep 07 11:28:51.306816 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:51.306364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv" event={"ID":"a76fc137-c23c-4d98-9b77-a769f24eafde","Type":"ContainerDied","Data":"b583cba332efa9a9511da12e08a78f8ccf67a11d686f3d6e3e888d63ae3d54fd"}
Sep 07 11:28:52.311831 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:52.311775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv" event={"ID":"a76fc137-c23c-4d98-9b77-a769f24eafde","Type":"ContainerStarted","Data":"7b97e51fc81a86ed8a088c07593756bab1f4f0ff7a675b09014e6e6203f6c425"}
Sep 07 11:28:52.312208 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:52.311948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:28:52.341920 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:28:52.341876    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv" podStartSLOduration=2.861902627 podStartE2EDuration="7.341860157s" totalImagesPullingTime="3.476373032s" totalInitContainerRuntime="1.003584498s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:28:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:28:52.340985555 +0000 UTC m=+382.046766245" watchObservedRunningTime="2026-09-07 11:28:52.341860157 +0000 UTC m=+382.047640846"
Sep 07 11:29:03.318711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:03.318674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f479d7474-bz4hv"
Sep 07 11:29:06.612934 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.612904    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"]
Sep 07 11:29:06.619810 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.619774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:06.622202 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.622182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 11:29:06.622285 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.622184    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 11:29:06.622345 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.622188    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-72d9d\""
Sep 07 11:29:06.627968 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.627947    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"]
Sep 07 11:29:06.700139 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.700109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e006b6-b789-4fc3-a699-1406354e6c58-tmp\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:06.700247 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.700147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e006b6-b789-4fc3-a699-1406354e6c58-tls-certs\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:06.700247 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.700183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgnp\" (UniqueName: \"kubernetes.io/projected/06e006b6-b789-4fc3-a699-1406354e6c58-kube-api-access-twgnp\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:06.800978 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.800955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e006b6-b789-4fc3-a699-1406354e6c58-tls-certs\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:06.801094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.800995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgnp\" (UniqueName: \"kubernetes.io/projected/06e006b6-b789-4fc3-a699-1406354e6c58-kube-api-access-twgnp\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:06.801094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.801044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e006b6-b789-4fc3-a699-1406354e6c58-tmp\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:06.801175 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:29:06.801107    2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 07 11:29:06.801214 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:29:06.801190    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e006b6-b789-4fc3-a699-1406354e6c58-tls-certs podName:06e006b6-b789-4fc3-a699-1406354e6c58 nodeName:}" failed. No retries permitted until 2026-09-07 11:29:07.301171194 +0000 UTC m=+397.006951865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06e006b6-b789-4fc3-a699-1406354e6c58-tls-certs") pod "kube-auth-proxy-5586fdf7b-tz4vp" (UID: "06e006b6-b789-4fc3-a699-1406354e6c58") : secret "kube-auth-proxy-tls" not found
Sep 07 11:29:06.803184 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.803163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e006b6-b789-4fc3-a699-1406354e6c58-tmp\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:06.810133 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:06.810113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgnp\" (UniqueName: \"kubernetes.io/projected/06e006b6-b789-4fc3-a699-1406354e6c58-kube-api-access-twgnp\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:07.304849 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.304783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e006b6-b789-4fc3-a699-1406354e6c58-tls-certs\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:07.307205 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.307186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e006b6-b789-4fc3-a699-1406354e6c58-tls-certs\") pod \"kube-auth-proxy-5586fdf7b-tz4vp\" (UID: \"06e006b6-b789-4fc3-a699-1406354e6c58\") " pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:07.530537 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.530504    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"
Sep 07 11:29:07.802578 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.802463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp"]
Sep 07 11:29:07.805529 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:29:07.805498    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e006b6_b789_4fc3_a699_1406354e6c58.slice/crio-156a738be52bf21aa516e1371f5b8df0b470953312ac5b537eccc21d8f0ba97d WatchSource:0}: Error finding container 156a738be52bf21aa516e1371f5b8df0b470953312ac5b537eccc21d8f0ba97d: Status 404 returned error can't find the container with id 156a738be52bf21aa516e1371f5b8df0b470953312ac5b537eccc21d8f0ba97d
Sep 07 11:29:07.913640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.913566    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"]
Sep 07 11:29:07.918046 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.918024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:07.920410 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.920386    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 07 11:29:07.920653 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.920636    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 07 11:29:07.920714 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.920651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 11:29:07.920962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.920945    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d7btp\""
Sep 07 11:29:07.921033 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.920946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 07 11:29:07.931473 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:07.931447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"]
Sep 07 11:29:08.009931 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.009899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.009931 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.009929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.010134 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.009968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.010134 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.010034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptm2\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-kube-api-access-2ptm2\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.010134 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.010066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.010134 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.010082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.010134 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.010096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.110808 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.110749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptm2\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-kube-api-access-2ptm2\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.110975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.110831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.110975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.110861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.110975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.110885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.110975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.110944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.110975 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.110969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.111231 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.111101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.111655 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.111630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.113568 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.113545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.113766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.113743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.113893 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.113858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.113942 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.113892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.118734 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.118712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.118930 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.118911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptm2\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-kube-api-access-2ptm2\") pod \"istiod-openshift-gateway-7b7995d5cd-cfmjw\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.227447 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.227417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:08.384064 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.384032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp" event={"ID":"06e006b6-b789-4fc3-a699-1406354e6c58","Type":"ContainerStarted","Data":"156a738be52bf21aa516e1371f5b8df0b470953312ac5b537eccc21d8f0ba97d"}
Sep 07 11:29:08.501675 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:08.501644    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"]
Sep 07 11:29:08.504339 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:29:08.504309    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3f30eb_d223_41b0_ae42_fcfaac10dbf6.slice/crio-aeb0fb1df5c77aec4cb27c0dc1550af9b02daa8b065c2749e0362a140cf3df4e WatchSource:0}: Error finding container aeb0fb1df5c77aec4cb27c0dc1550af9b02daa8b065c2749e0362a140cf3df4e: Status 404 returned error can't find the container with id aeb0fb1df5c77aec4cb27c0dc1550af9b02daa8b065c2749e0362a140cf3df4e
Sep 07 11:29:09.390250 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:09.390178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw" event={"ID":"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6","Type":"ContainerStarted","Data":"aeb0fb1df5c77aec4cb27c0dc1550af9b02daa8b065c2749e0362a140cf3df4e"}
Sep 07 11:29:11.402111 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:11.402072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp" event={"ID":"06e006b6-b789-4fc3-a699-1406354e6c58","Type":"ContainerStarted","Data":"661b3444241d7e8724e2785b3d3af14c3cc9fdd24ea5ff57f57a3e16f11cdc00"}
Sep 07 11:29:11.421755 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:11.421702    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5586fdf7b-tz4vp" podStartSLOduration=2.745119989 podStartE2EDuration="5.421685267s" totalImagesPullingTime="2.676565278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:29:11.420004494 +0000 UTC m=+401.125785196" watchObservedRunningTime="2026-09-07 11:29:11.421685267 +0000 UTC m=+401.127465956"
Sep 07 11:29:12.200673 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:12.200614    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 11:29:12.200787 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:12.200693    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 11:29:12.407600 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:12.407562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw" event={"ID":"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6","Type":"ContainerStarted","Data":"e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f"}
Sep 07 11:29:12.429578 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:12.429523    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw" podStartSLOduration=1.736251802 podStartE2EDuration="5.429509051s" totalImagesPullingTime="3.693257249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:29:12.427318357 +0000 UTC m=+402.133099047" watchObservedRunningTime="2026-09-07 11:29:12.429509051 +0000 UTC m=+402.135289741"
Sep 07 11:29:13.411980 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:13.411950    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:13.413429 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:13.413396    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cfmjw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 11:29:13.413555 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:13.413460    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw" podUID="3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 11:29:14.417413 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:14.417387    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:25.116469 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.116427    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"]
Sep 07 11:29:25.119831 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.119811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.132563 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.132536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"]
Sep 07 11:29:25.247906 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.247870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3258b698-e714-4e3f-a3be-09039586bd22-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.248075 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.247916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3258b698-e714-4e3f-a3be-09039586bd22-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.248075 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.247938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.248075 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.248005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.248075 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.248027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3258b698-e714-4e3f-a3be-09039586bd22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.248075 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.248058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.248075 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.248079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnwpq\" (UniqueName: \"kubernetes.io/projected/3258b698-e714-4e3f-a3be-09039586bd22-kube-api-access-pnwpq\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.348984 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.348949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3258b698-e714-4e3f-a3be-09039586bd22-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.349171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.349001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3258b698-e714-4e3f-a3be-09039586bd22-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.349171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.349033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.349171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.349070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.349171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.349098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3258b698-e714-4e3f-a3be-09039586bd22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.349171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.349138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.349171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.349168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnwpq\" (UniqueName: \"kubernetes.io/projected/3258b698-e714-4e3f-a3be-09039586bd22-kube-api-access-pnwpq\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.349971 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.349942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3258b698-e714-4e3f-a3be-09039586bd22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.351485 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.351459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.351597 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.351532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.351660 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.351639    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3258b698-e714-4e3f-a3be-09039586bd22-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.351696 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.351646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3258b698-e714-4e3f-a3be-09039586bd22-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.363512 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.363487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3258b698-e714-4e3f-a3be-09039586bd22-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.368243 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.368191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnwpq\" (UniqueName: \"kubernetes.io/projected/3258b698-e714-4e3f-a3be-09039586bd22-kube-api-access-pnwpq\") pod \"istiod-openshift-gateway-5677ccb49b-nrvfw\" (UID: \"3258b698-e714-4e3f-a3be-09039586bd22\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.428264 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.428235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:25.703814 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.703719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"]
Sep 07 11:29:25.706819 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:29:25.706779    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3258b698_e714_4e3f_a3be_09039586bd22.slice/crio-948c68211ce31884898c795fbd063a06c03ad85742b4ed677cb0afa4615ba5cf WatchSource:0}: Error finding container 948c68211ce31884898c795fbd063a06c03ad85742b4ed677cb0afa4615ba5cf: Status 404 returned error can't find the container with id 948c68211ce31884898c795fbd063a06c03ad85742b4ed677cb0afa4615ba5cf
Sep 07 11:29:25.709677 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.709640    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 11:29:25.709770 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:25.709701    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 11:29:26.468605 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.468566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw" event={"ID":"3258b698-e714-4e3f-a3be-09039586bd22","Type":"ContainerStarted","Data":"e92b305d3670db5239e63e0be9734629267ec4b96762ba63237c936837c0bcf5"}
Sep 07 11:29:26.469035 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.468619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw" event={"ID":"3258b698-e714-4e3f-a3be-09039586bd22","Type":"ContainerStarted","Data":"948c68211ce31884898c795fbd063a06c03ad85742b4ed677cb0afa4615ba5cf"}
Sep 07 11:29:26.469035 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.468815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:26.470387 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.470367    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw"
Sep 07 11:29:26.491743 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.491681    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nrvfw" podStartSLOduration=1.491666601 podStartE2EDuration="1.491666601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:29:26.491431565 +0000 UTC m=+416.197212256" watchObservedRunningTime="2026-09-07 11:29:26.491666601 +0000 UTC m=+416.197447290"
Sep 07 11:29:26.544196 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.544165    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"]
Sep 07 11:29:26.544573 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.544357    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw" podUID="3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" containerName="discovery" containerID="cri-o://e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f" gracePeriod=30
Sep 07 11:29:26.786670 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.786647    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:26.860249 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.860223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-kubeconfig\") pod \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") "
Sep 07 11:29:26.860376 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.860261    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-dns-cert\") pod \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") "
Sep 07 11:29:26.860376 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.860289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-local-certs\") pod \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") "
Sep 07 11:29:26.860376 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.860313    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-token\") pod \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") "
Sep 07 11:29:26.860547 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.860391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-ca-configmap\") pod \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") "
Sep 07 11:29:26.860547 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.860424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-cacerts\") pod \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") "
Sep 07 11:29:26.860547 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.860493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ptm2\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-kube-api-access-2ptm2\") pod \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\" (UID: \"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6\") "
Sep 07 11:29:26.860825 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.860777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" (UID: "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:29:26.862673 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.862641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" (UID: "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:29:26.862770 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.862650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" (UID: "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:29:26.862978 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.862953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-local-certs" (OuterVolumeSpecName: "local-certs") pod "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" (UID: "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:29:26.862978 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.862961    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" (UID: "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:29:26.863112 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.863015    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-kube-api-access-2ptm2" (OuterVolumeSpecName: "kube-api-access-2ptm2") pod "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" (UID: "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6"). InnerVolumeSpecName "kube-api-access-2ptm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:29:26.863112 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.863075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-cacerts" (OuterVolumeSpecName: "cacerts") pod "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" (UID: "3e3f30eb-d223-41b0-ae42-fcfaac10dbf6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:29:26.961957 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.961931    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-ca-configmap\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:29:26.961957 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.961956    2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-cacerts\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:29:26.962140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.961970    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ptm2\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-kube-api-access-2ptm2\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:29:26.962140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.961984    2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-kubeconfig\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:29:26.962140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.962042    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-csr-dns-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:29:26.962140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.962055    2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-local-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:29:26.962140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:26.962067    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:29:27.474591 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.474554    2581 generic.go:356] "Generic (PLEG): container finished" podID="3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" containerID="e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f" exitCode=0
Sep 07 11:29:27.475089 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.474606    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"
Sep 07 11:29:27.475089 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.474612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw" event={"ID":"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6","Type":"ContainerDied","Data":"e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f"}
Sep 07 11:29:27.475089 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.474666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw" event={"ID":"3e3f30eb-d223-41b0-ae42-fcfaac10dbf6","Type":"ContainerDied","Data":"aeb0fb1df5c77aec4cb27c0dc1550af9b02daa8b065c2749e0362a140cf3df4e"}
Sep 07 11:29:27.475089 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.474688    2581 scope.go:117] "RemoveContainer" containerID="e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f"
Sep 07 11:29:27.483044 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.483026    2581 scope.go:117] "RemoveContainer" containerID="e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f"
Sep 07 11:29:27.483306 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:29:27.483286    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f\": container with ID starting with e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f not found: ID does not exist" containerID="e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f"
Sep 07 11:29:27.483380 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.483319    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f"} err="failed to get container status \"e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f\": rpc error: code = NotFound desc = could not find container \"e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f\": container with ID starting with e8d4ebd1f809822e2294efa80a6ef22701d3fac420d2af4650817891a1bea98f not found: ID does not exist"
Sep 07 11:29:27.508447 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.508412    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"]
Sep 07 11:29:27.517348 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.517323    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfmjw"]
Sep 07 11:29:27.720332 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.720302    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"]
Sep 07 11:29:27.720635 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.720622    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" containerName="discovery"
Sep 07 11:29:27.720677 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.720637    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" containerName="discovery"
Sep 07 11:29:27.720710 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.720687    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" containerName="discovery"
Sep 07 11:29:27.729827 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.729745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:27.732291 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.732263    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"]
Sep 07 11:29:27.732673 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.732652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 07 11:29:27.732763 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.732653    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8mq8m\""
Sep 07 11:29:27.735385 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.735364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"]
Sep 07 11:29:27.735490 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.735464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:27.737844 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.737827    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 07 11:29:27.738351 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.737907    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gpk6c\""
Sep 07 11:29:27.745629 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.745608    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"]
Sep 07 11:29:27.769181 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.769155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert\") pod \"kserve-controller-manager-d6f4b65b8-pdwnc\" (UID: \"7c6de5af-c4ca-4006-b860-8269604c34c7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:27.769281 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.769259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl2tg\" (UniqueName: \"kubernetes.io/projected/7c6de5af-c4ca-4006-b860-8269604c34c7-kube-api-access-pl2tg\") pod \"kserve-controller-manager-d6f4b65b8-pdwnc\" (UID: \"7c6de5af-c4ca-4006-b860-8269604c34c7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:27.870067 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.870034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjb2\" (UniqueName: \"kubernetes.io/projected/64798131-40bb-44d6-8461-808df5e5e9cf-kube-api-access-rwjb2\") pod \"llmisvc-controller-manager-777c6bd7d5-dg9rw\" (UID: \"64798131-40bb-44d6-8461-808df5e5e9cf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:27.870205 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.870073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert\") pod \"kserve-controller-manager-d6f4b65b8-pdwnc\" (UID: \"7c6de5af-c4ca-4006-b860-8269604c34c7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:27.870205 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.870094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64798131-40bb-44d6-8461-808df5e5e9cf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dg9rw\" (UID: \"64798131-40bb-44d6-8461-808df5e5e9cf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:27.870205 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.870189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl2tg\" (UniqueName: \"kubernetes.io/projected/7c6de5af-c4ca-4006-b860-8269604c34c7-kube-api-access-pl2tg\") pod \"kserve-controller-manager-d6f4b65b8-pdwnc\" (UID: \"7c6de5af-c4ca-4006-b860-8269604c34c7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:27.870355 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:29:27.870211    2581 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 07 11:29:27.870355 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:29:27.870269    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert podName:7c6de5af-c4ca-4006-b860-8269604c34c7 nodeName:}" failed. No retries permitted until 2026-09-07 11:29:28.370248593 +0000 UTC m=+418.076029261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert") pod "kserve-controller-manager-d6f4b65b8-pdwnc" (UID: "7c6de5af-c4ca-4006-b860-8269604c34c7") : secret "kserve-webhook-server-cert" not found
Sep 07 11:29:27.890510 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.890483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl2tg\" (UniqueName: \"kubernetes.io/projected/7c6de5af-c4ca-4006-b860-8269604c34c7-kube-api-access-pl2tg\") pod \"kserve-controller-manager-d6f4b65b8-pdwnc\" (UID: \"7c6de5af-c4ca-4006-b860-8269604c34c7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:27.971551 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.971514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjb2\" (UniqueName: \"kubernetes.io/projected/64798131-40bb-44d6-8461-808df5e5e9cf-kube-api-access-rwjb2\") pod \"llmisvc-controller-manager-777c6bd7d5-dg9rw\" (UID: \"64798131-40bb-44d6-8461-808df5e5e9cf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:27.971706 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.971564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64798131-40bb-44d6-8461-808df5e5e9cf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dg9rw\" (UID: \"64798131-40bb-44d6-8461-808df5e5e9cf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:27.973869 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.973850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64798131-40bb-44d6-8461-808df5e5e9cf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dg9rw\" (UID: \"64798131-40bb-44d6-8461-808df5e5e9cf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:27.986529 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:27.986472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjb2\" (UniqueName: \"kubernetes.io/projected/64798131-40bb-44d6-8461-808df5e5e9cf-kube-api-access-rwjb2\") pod \"llmisvc-controller-manager-777c6bd7d5-dg9rw\" (UID: \"64798131-40bb-44d6-8461-808df5e5e9cf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:28.050611 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:28.050587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:28.320718 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:28.320644    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"]
Sep 07 11:29:28.323267 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:29:28.323243    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod64798131_40bb_44d6_8461_808df5e5e9cf.slice/crio-e54e76161fa7b313932acf8f275879b60632728687c8acefbcf4b8108032ed27 WatchSource:0}: Error finding container e54e76161fa7b313932acf8f275879b60632728687c8acefbcf4b8108032ed27: Status 404 returned error can't find the container with id e54e76161fa7b313932acf8f275879b60632728687c8acefbcf4b8108032ed27
Sep 07 11:29:28.374740 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:28.374709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert\") pod \"kserve-controller-manager-d6f4b65b8-pdwnc\" (UID: \"7c6de5af-c4ca-4006-b860-8269604c34c7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:28.374852 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:29:28.374841    2581 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 07 11:29:28.374907 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:29:28.374896    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert podName:7c6de5af-c4ca-4006-b860-8269604c34c7 nodeName:}" failed. No retries permitted until 2026-09-07 11:29:29.374877438 +0000 UTC m=+419.080658107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert") pod "kserve-controller-manager-d6f4b65b8-pdwnc" (UID: "7c6de5af-c4ca-4006-b860-8269604c34c7") : secret "kserve-webhook-server-cert" not found
Sep 07 11:29:28.479561 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:28.479524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw" event={"ID":"64798131-40bb-44d6-8461-808df5e5e9cf","Type":"ContainerStarted","Data":"e54e76161fa7b313932acf8f275879b60632728687c8acefbcf4b8108032ed27"}
Sep 07 11:29:28.939886 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:28.939851    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e3f30eb-d223-41b0-ae42-fcfaac10dbf6" path="/var/lib/kubelet/pods/3e3f30eb-d223-41b0-ae42-fcfaac10dbf6/volumes"
Sep 07 11:29:29.383234 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:29.383202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert\") pod \"kserve-controller-manager-d6f4b65b8-pdwnc\" (UID: \"7c6de5af-c4ca-4006-b860-8269604c34c7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:29.385396 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:29.385368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c6de5af-c4ca-4006-b860-8269604c34c7-cert\") pod \"kserve-controller-manager-d6f4b65b8-pdwnc\" (UID: \"7c6de5af-c4ca-4006-b860-8269604c34c7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:29.544414 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:29.544383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:29.868445 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:29.868411    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"]
Sep 07 11:29:29.870189 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:29:29.870149    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6de5af_c4ca_4006_b860_8269604c34c7.slice/crio-6430c885edeea6e1f9a525303064e019b9362d370b30498d7af696aedcde0402 WatchSource:0}: Error finding container 6430c885edeea6e1f9a525303064e019b9362d370b30498d7af696aedcde0402: Status 404 returned error can't find the container with id 6430c885edeea6e1f9a525303064e019b9362d370b30498d7af696aedcde0402
Sep 07 11:29:30.491414 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:30.491376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc" event={"ID":"7c6de5af-c4ca-4006-b860-8269604c34c7","Type":"ContainerStarted","Data":"6430c885edeea6e1f9a525303064e019b9362d370b30498d7af696aedcde0402"}
Sep 07 11:29:31.496645 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:31.496603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw" event={"ID":"64798131-40bb-44d6-8461-808df5e5e9cf","Type":"ContainerStarted","Data":"4c5824af80166a8a4d9689cf2702aa1a66730ade0f8b8ba651cefd5532e6920d"}
Sep 07 11:29:31.497052 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:31.496701    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:29:31.515506 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:31.515455    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw" podStartSLOduration=1.952893015 podStartE2EDuration="4.515440321s" totalImagesPullingTime="2.562547306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:29:31.513349827 +0000 UTC m=+421.219130530" watchObservedRunningTime="2026-09-07 11:29:31.515440321 +0000 UTC m=+421.221221008"
Sep 07 11:29:32.502554 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:32.502523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc" event={"ID":"7c6de5af-c4ca-4006-b860-8269604c34c7","Type":"ContainerStarted","Data":"adfb7bd2907fde547369ae6be3c764f46a33cdf09b6cb43c2e1c4019cda1cd8b"}
Sep 07 11:29:32.502868 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:32.502743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:29:32.520847 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:29:32.520779    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc" podStartSLOduration=2.959044259 podStartE2EDuration="5.520761507s" totalImagesPullingTime="2.561717248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:29:32.519326721 +0000 UTC m=+422.225107414" watchObservedRunningTime="2026-09-07 11:29:32.520761507 +0000 UTC m=+422.226542198"
Sep 07 11:30:02.504847 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:02.504757    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dg9rw"
Sep 07 11:30:03.512117 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:03.512086    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdwnc"
Sep 07 11:30:49.320246 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.320216    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-r5c8l"]
Sep 07 11:30:49.323543 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.323526    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l"
Sep 07 11:30:49.327214 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.327195    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 11:30:49.328046 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.328025    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ztwc9\""
Sep 07 11:30:49.328296 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.328279    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 11:30:49.345423 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.345402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-r5c8l"]
Sep 07 11:30:49.435986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.435951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstxg\" (UniqueName: \"kubernetes.io/projected/87974954-50b8-4fe4-95fa-17dd6c1f2c39-kube-api-access-nstxg\") pod \"authorino-operator-68d88948d8-r5c8l\" (UID: \"87974954-50b8-4fe4-95fa-17dd6c1f2c39\") " pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l"
Sep 07 11:30:49.536392 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.536358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstxg\" (UniqueName: \"kubernetes.io/projected/87974954-50b8-4fe4-95fa-17dd6c1f2c39-kube-api-access-nstxg\") pod \"authorino-operator-68d88948d8-r5c8l\" (UID: \"87974954-50b8-4fe4-95fa-17dd6c1f2c39\") " pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l"
Sep 07 11:30:49.546396 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.546371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstxg\" (UniqueName: \"kubernetes.io/projected/87974954-50b8-4fe4-95fa-17dd6c1f2c39-kube-api-access-nstxg\") pod \"authorino-operator-68d88948d8-r5c8l\" (UID: \"87974954-50b8-4fe4-95fa-17dd6c1f2c39\") " pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l"
Sep 07 11:30:49.633669 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.633601    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l"
Sep 07 11:30:49.911471 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:49.911442    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-r5c8l"]
Sep 07 11:30:49.913551 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:30:49.913521    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87974954_50b8_4fe4_95fa_17dd6c1f2c39.slice/crio-217145eb958ebde4e8f999281fe0e8e9c1db9e52559c01ed961db4f024e675c2 WatchSource:0}: Error finding container 217145eb958ebde4e8f999281fe0e8e9c1db9e52559c01ed961db4f024e675c2: Status 404 returned error can't find the container with id 217145eb958ebde4e8f999281fe0e8e9c1db9e52559c01ed961db4f024e675c2
Sep 07 11:30:50.832278 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:50.832232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l" event={"ID":"87974954-50b8-4fe4-95fa-17dd6c1f2c39","Type":"ContainerStarted","Data":"217145eb958ebde4e8f999281fe0e8e9c1db9e52559c01ed961db4f024e675c2"}
Sep 07 11:30:52.842608 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:52.842572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l" event={"ID":"87974954-50b8-4fe4-95fa-17dd6c1f2c39","Type":"ContainerStarted","Data":"0d56ce25a3838e782ab12e6df428e22825e90fc1a266fe1e65c813291bd86c9a"}
Sep 07 11:30:52.843079 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:52.842710    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l"
Sep 07 11:30:52.873679 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:30:52.873629    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l" podStartSLOduration=1.4995288279999999 podStartE2EDuration="3.873613299s" totalImagesPullingTime="2.374084471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:30:52.870909714 +0000 UTC m=+502.576690403" watchObservedRunningTime="2026-09-07 11:30:52.873613299 +0000 UTC m=+502.579393989"
Sep 07 11:31:03.849705 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:03.849671    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-r5c8l"
Sep 07 11:31:05.252870 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.252830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"]
Sep 07 11:31:05.255986 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.255971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:05.258253 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.258234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvcp\" (UniqueName: \"kubernetes.io/projected/cff07c40-5731-4e52-9fcc-87c735a9ca06-kube-api-access-tjvcp\") pod \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" (UID: \"cff07c40-5731-4e52-9fcc-87c735a9ca06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:05.258325 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.258267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cff07c40-5731-4e52-9fcc-87c735a9ca06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" (UID: \"cff07c40-5731-4e52-9fcc-87c735a9ca06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:05.258769 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.258752    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6brmr\""
Sep 07 11:31:05.269895 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.269863    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"]
Sep 07 11:31:05.359229 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.359187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvcp\" (UniqueName: \"kubernetes.io/projected/cff07c40-5731-4e52-9fcc-87c735a9ca06-kube-api-access-tjvcp\") pod \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" (UID: \"cff07c40-5731-4e52-9fcc-87c735a9ca06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:05.359387 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.359240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cff07c40-5731-4e52-9fcc-87c735a9ca06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" (UID: \"cff07c40-5731-4e52-9fcc-87c735a9ca06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:05.359646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.359624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cff07c40-5731-4e52-9fcc-87c735a9ca06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" (UID: \"cff07c40-5731-4e52-9fcc-87c735a9ca06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:05.370584 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.370555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvcp\" (UniqueName: \"kubernetes.io/projected/cff07c40-5731-4e52-9fcc-87c735a9ca06-kube-api-access-tjvcp\") pod \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" (UID: \"cff07c40-5731-4e52-9fcc-87c735a9ca06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:05.565598 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.565519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:05.838032 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.837368    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"]
Sep 07 11:31:05.853122 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:05.852736    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"]
Sep 07 11:31:05.968667 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:31:05.968629    2581 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 07 11:31:05.968667 ip-10-0-143-241 kubenswrapper[2581]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-68cf589589-zmd2p_kuadrant-system_cff07c40-5731-4e52-9fcc-87c735a9ca06_0(04737913d5d1d32971ce7c56760c22e0a07ab55cf920db0f96ad05f1c5d32a63): error adding pod kuadrant-system_kuadrant-operator-controller-manager-68cf589589-zmd2p to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"04737913d5d1d32971ce7c56760c22e0a07ab55cf920db0f96ad05f1c5d32a63" Netns:"/var/run/netns/e362de74-0c01-4cf5-a59c-a88078dd3aba" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-68cf589589-zmd2p;K8S_POD_INFRA_CONTAINER_ID=04737913d5d1d32971ce7c56760c22e0a07ab55cf920db0f96ad05f1c5d32a63;K8S_POD_UID=cff07c40-5731-4e52-9fcc-87c735a9ca06" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p/cff07c40-5731-4e52-9fcc-87c735a9ca06]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-68cf589589-zmd2p in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-68cf589589-zmd2p in out of cluster comm: status update failed for pod kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p: pods "kuadrant-operator-controller-manager-68cf589589-zmd2p" not found
Sep 07 11:31:05.968667 ip-10-0-143-241 kubenswrapper[2581]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 07 11:31:05.968667 ip-10-0-143-241 kubenswrapper[2581]:  >
Sep 07 11:31:05.968900 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:31:05.968701    2581 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 07 11:31:05.968900 ip-10-0-143-241 kubenswrapper[2581]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-68cf589589-zmd2p_kuadrant-system_cff07c40-5731-4e52-9fcc-87c735a9ca06_0(04737913d5d1d32971ce7c56760c22e0a07ab55cf920db0f96ad05f1c5d32a63): error adding pod kuadrant-system_kuadrant-operator-controller-manager-68cf589589-zmd2p to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"04737913d5d1d32971ce7c56760c22e0a07ab55cf920db0f96ad05f1c5d32a63" Netns:"/var/run/netns/e362de74-0c01-4cf5-a59c-a88078dd3aba" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-68cf589589-zmd2p;K8S_POD_INFRA_CONTAINER_ID=04737913d5d1d32971ce7c56760c22e0a07ab55cf920db0f96ad05f1c5d32a63;K8S_POD_UID=cff07c40-5731-4e52-9fcc-87c735a9ca06" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p/cff07c40-5731-4e52-9fcc-87c735a9ca06]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-68cf589589-zmd2p in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-68cf589589-zmd2p in out of cluster comm: status update failed for pod kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p: pods "kuadrant-operator-controller-manager-68cf589589-zmd2p" not found
Sep 07 11:31:05.968900 ip-10-0-143-241 kubenswrapper[2581]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 07 11:31:05.968900 ip-10-0-143-241 kubenswrapper[2581]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:06.907482 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:06.907451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:06.909711 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:31:06.909682    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="cff07c40-5731-4e52-9fcc-87c735a9ca06" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:06.915245 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:06.915225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:06.917670 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:31:06.917646    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="cff07c40-5731-4e52-9fcc-87c735a9ca06" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:06.973564 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:06.973533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cff07c40-5731-4e52-9fcc-87c735a9ca06-extensions-socket-volume\") pod \"cff07c40-5731-4e52-9fcc-87c735a9ca06\" (UID: \"cff07c40-5731-4e52-9fcc-87c735a9ca06\") "
Sep 07 11:31:06.973691 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:06.973636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjvcp\" (UniqueName: \"kubernetes.io/projected/cff07c40-5731-4e52-9fcc-87c735a9ca06-kube-api-access-tjvcp\") pod \"cff07c40-5731-4e52-9fcc-87c735a9ca06\" (UID: \"cff07c40-5731-4e52-9fcc-87c735a9ca06\") "
Sep 07 11:31:06.973824 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:06.973781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff07c40-5731-4e52-9fcc-87c735a9ca06-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cff07c40-5731-4e52-9fcc-87c735a9ca06" (UID: "cff07c40-5731-4e52-9fcc-87c735a9ca06"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:31:06.975618 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:06.975585    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff07c40-5731-4e52-9fcc-87c735a9ca06-kube-api-access-tjvcp" (OuterVolumeSpecName: "kube-api-access-tjvcp") pod "cff07c40-5731-4e52-9fcc-87c735a9ca06" (UID: "cff07c40-5731-4e52-9fcc-87c735a9ca06"). InnerVolumeSpecName "kube-api-access-tjvcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:31:07.074305 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:07.074266    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvcp\" (UniqueName: \"kubernetes.io/projected/cff07c40-5731-4e52-9fcc-87c735a9ca06-kube-api-access-tjvcp\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:31:07.074305 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:07.074297    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cff07c40-5731-4e52-9fcc-87c735a9ca06-extensions-socket-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:31:07.911824 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:07.911778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:07.920134 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:31:07.920097    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-zmd2p\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="cff07c40-5731-4e52-9fcc-87c735a9ca06" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-zmd2p"
Sep 07 11:31:08.940527 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:08.940493    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cff07c40-5731-4e52-9fcc-87c735a9ca06" path="/var/lib/kubelet/pods/cff07c40-5731-4e52-9fcc-87c735a9ca06/volumes"
Sep 07 11:31:21.674444 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.674406    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"]
Sep 07 11:31:21.680861 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.680814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:21.683444 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.683416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6brmr\""
Sep 07 11:31:21.690362 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.690335    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"]
Sep 07 11:31:21.789106 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.789075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcvf5\" (UniqueName: \"kubernetes.io/projected/cc6a6c66-86c6-4263-93bf-f73258c01180-kube-api-access-tcvf5\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-mb6pl\" (UID: \"cc6a6c66-86c6-4263-93bf-f73258c01180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:21.789241 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.789114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc6a6c66-86c6-4263-93bf-f73258c01180-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-mb6pl\" (UID: \"cc6a6c66-86c6-4263-93bf-f73258c01180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:21.889711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.889677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcvf5\" (UniqueName: \"kubernetes.io/projected/cc6a6c66-86c6-4263-93bf-f73258c01180-kube-api-access-tcvf5\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-mb6pl\" (UID: \"cc6a6c66-86c6-4263-93bf-f73258c01180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:21.889866 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.889719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc6a6c66-86c6-4263-93bf-f73258c01180-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-mb6pl\" (UID: \"cc6a6c66-86c6-4263-93bf-f73258c01180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:21.890065 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.890050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc6a6c66-86c6-4263-93bf-f73258c01180-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-mb6pl\" (UID: \"cc6a6c66-86c6-4263-93bf-f73258c01180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:21.898038 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.898011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcvf5\" (UniqueName: \"kubernetes.io/projected/cc6a6c66-86c6-4263-93bf-f73258c01180-kube-api-access-tcvf5\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-mb6pl\" (UID: \"cc6a6c66-86c6-4263-93bf-f73258c01180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:21.991330 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:21.991299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:22.268586 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:22.268561    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"]
Sep 07 11:31:22.270826 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:31:22.270781    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6a6c66_86c6_4263_93bf_f73258c01180.slice/crio-ca9cd840587b59831d6c82ff8ab9e0259729f72df380c362a4188ebecdaa271a WatchSource:0}: Error finding container ca9cd840587b59831d6c82ff8ab9e0259729f72df380c362a4188ebecdaa271a: Status 404 returned error can't find the container with id ca9cd840587b59831d6c82ff8ab9e0259729f72df380c362a4188ebecdaa271a
Sep 07 11:31:22.979700 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:22.979660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl" event={"ID":"cc6a6c66-86c6-4263-93bf-f73258c01180","Type":"ContainerStarted","Data":"ca9cd840587b59831d6c82ff8ab9e0259729f72df380c362a4188ebecdaa271a"}
Sep 07 11:31:25.997701 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:25.997663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl" event={"ID":"cc6a6c66-86c6-4263-93bf-f73258c01180","Type":"ContainerStarted","Data":"97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437"}
Sep 07 11:31:25.998110 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:25.997815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:31:26.019612 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:26.019551    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl" podStartSLOduration=1.445443748 podStartE2EDuration="5.01953807s" totalImagesPullingTime="3.574094322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:31:26.018168342 +0000 UTC m=+535.723949032" watchObservedRunningTime="2026-09-07 11:31:26.01953807 +0000 UTC m=+535.725318759"
Sep 07 11:31:37.004139 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:31:37.004106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:32:32.314837 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.314786    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f74dc8dd5-gpqwd"]
Sep 07 11:32:32.318411 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.318387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.321133 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.321114    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 11:32:32.321225 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.321139    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 11:32:32.322174 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.322155    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4pz7p\""
Sep 07 11:32:32.322383 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.322368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 11:32:32.328540 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.328519    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f74dc8dd5-gpqwd"]
Sep 07 11:32:32.431911 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.431878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-webhook-cert\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.432066 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.431934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-metrics-certs\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.432066 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.431990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wkvx\" (UniqueName: \"kubernetes.io/projected/05b120ff-2097-407b-a942-106d280f99cb-kube-api-access-9wkvx\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.532541 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.532508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-metrics-certs\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.532681 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.532549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wkvx\" (UniqueName: \"kubernetes.io/projected/05b120ff-2097-407b-a942-106d280f99cb-kube-api-access-9wkvx\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.532681 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.532668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-webhook-cert\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.534927 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.534906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-webhook-cert\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.535024 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.534956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-metrics-certs\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.541391 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.541366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wkvx\" (UniqueName: \"kubernetes.io/projected/05b120ff-2097-407b-a942-106d280f99cb-kube-api-access-9wkvx\") pod \"maas-controller-f74dc8dd5-gpqwd\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") " pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.630004 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.629918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:32.901229 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:32.901147    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f74dc8dd5-gpqwd"]
Sep 07 11:32:32.904184 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:32:32.904150    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b120ff_2097_407b_a942_106d280f99cb.slice/crio-68fda04ec833c2d7e21ce1f6b73d77ee10654cccb8eb5d2273d16d6b58ec7f1b WatchSource:0}: Error finding container 68fda04ec833c2d7e21ce1f6b73d77ee10654cccb8eb5d2273d16d6b58ec7f1b: Status 404 returned error can't find the container with id 68fda04ec833c2d7e21ce1f6b73d77ee10654cccb8eb5d2273d16d6b58ec7f1b
Sep 07 11:32:33.293656 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:33.293623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd" event={"ID":"05b120ff-2097-407b-a942-106d280f99cb","Type":"ContainerStarted","Data":"68fda04ec833c2d7e21ce1f6b73d77ee10654cccb8eb5d2273d16d6b58ec7f1b"}
Sep 07 11:32:35.304949 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:35.304919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd" event={"ID":"05b120ff-2097-407b-a942-106d280f99cb","Type":"ContainerStarted","Data":"502229a439276a07a905016a60b99ccb3a42acf6be017099fe83a5463fa98363"}
Sep 07 11:32:35.305270 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:35.305017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:32:35.327507 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:35.327464    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd" podStartSLOduration=1.021308572 podStartE2EDuration="3.327448313s" totalImagesPullingTime="2.306139741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:32:35.326279651 +0000 UTC m=+605.032060337" watchObservedRunningTime="2026-09-07 11:32:35.327448313 +0000 UTC m=+605.033229004"
Sep 07 11:32:46.311245 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:32:46.311210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:33:09.011457 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.011424    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"]
Sep 07 11:33:09.019589 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.019573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.024675 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.024639    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 11:33:09.024675 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.024655    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gtjll\""
Sep 07 11:33:09.024886 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.024746    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 11:33:09.024886 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.024758    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 11:33:09.024886 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.024746    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 11:33:09.037782 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.037761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"]
Sep 07 11:33:09.145592 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.145561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gbh\" (UniqueName: \"kubernetes.io/projected/529dc0e4-9e4e-4294-a202-2d292fae5c5f-kube-api-access-s4gbh\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.145762 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.145624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-maas-api-tls\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.145762 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.145644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529dc0e4-9e4e-4294-a202-2d292fae5c5f-openshift-service-ca\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.145890 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.145769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-metrics-certs\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.246558 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.246522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-metrics-certs\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.246558 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.246562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gbh\" (UniqueName: \"kubernetes.io/projected/529dc0e4-9e4e-4294-a202-2d292fae5c5f-kube-api-access-s4gbh\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.246804 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.246746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-maas-api-tls\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.246804 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.246781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529dc0e4-9e4e-4294-a202-2d292fae5c5f-openshift-service-ca\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.247398 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.247377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529dc0e4-9e4e-4294-a202-2d292fae5c5f-openshift-service-ca\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.249049 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.249028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-maas-api-tls\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.249191 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.249129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-metrics-certs\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.255555 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.255537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gbh\" (UniqueName: \"kubernetes.io/projected/529dc0e4-9e4e-4294-a202-2d292fae5c5f-kube-api-access-s4gbh\") pod \"maas-api-79f7b4565f-dswq9\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") " pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.331196 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.331130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:09.616176 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.616075    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"]
Sep 07 11:33:09.618756 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:33:09.618727    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529dc0e4_9e4e_4294_a202_2d292fae5c5f.slice/crio-1fe70ae8fc813f2cde5c8a37c3f5080c47b4f0f63cb1c87fbf99e3ad951dc6e9 WatchSource:0}: Error finding container 1fe70ae8fc813f2cde5c8a37c3f5080c47b4f0f63cb1c87fbf99e3ad951dc6e9: Status 404 returned error can't find the container with id 1fe70ae8fc813f2cde5c8a37c3f5080c47b4f0f63cb1c87fbf99e3ad951dc6e9
Sep 07 11:33:09.620296 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:09.620279    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:33:10.470806 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:10.470748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9" event={"ID":"529dc0e4-9e4e-4294-a202-2d292fae5c5f","Type":"ContainerStarted","Data":"1fe70ae8fc813f2cde5c8a37c3f5080c47b4f0f63cb1c87fbf99e3ad951dc6e9"}
Sep 07 11:33:11.476923 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:11.476884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9" event={"ID":"529dc0e4-9e4e-4294-a202-2d292fae5c5f","Type":"ContainerStarted","Data":"ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05"}
Sep 07 11:33:11.477304 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:11.477028    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:11.497052 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:11.497000    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9" podStartSLOduration=1.9371422630000001 podStartE2EDuration="3.496952236s" totalImagesPullingTime="1.559809973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:33:11.495476399 +0000 UTC m=+641.201257088" watchObservedRunningTime="2026-09-07 11:33:11.496952236 +0000 UTC m=+641.202732926"
Sep 07 11:33:17.486376 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:17.486346    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:33:30.298801 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.298747    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"]
Sep 07 11:33:30.302804 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.302774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.306049 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.306023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 07 11:33:30.306049 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.306036    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 11:33:30.306049 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.306025    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bf2b6\""
Sep 07 11:33:30.306231 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.306086    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 11:33:30.311568 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.311546    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"]
Sep 07 11:33:30.416510 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.416471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftslb\" (UniqueName: \"kubernetes.io/projected/7566aea8-4484-418c-9887-486f12ac16ec-kube-api-access-ftslb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.416510 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.416515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.416775 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.416538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.416775 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.416553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.416775 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.416663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7566aea8-4484-418c-9887-486f12ac16ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.416775 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.416744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517451 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517451 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517658 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517658 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7566aea8-4484-418c-9887-486f12ac16ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517658 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517658 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftslb\" (UniqueName: \"kubernetes.io/projected/7566aea8-4484-418c-9887-486f12ac16ec-kube-api-access-ftslb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517913 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.517972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.517957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.519772 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.519749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7566aea8-4484-418c-9887-486f12ac16ec-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.520133 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.520117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7566aea8-4484-418c-9887-486f12ac16ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.526432 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.526411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftslb\" (UniqueName: \"kubernetes.io/projected/7566aea8-4484-418c-9887-486f12ac16ec-kube-api-access-ftslb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8l9r2\" (UID: \"7566aea8-4484-418c-9887-486f12ac16ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.613761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.613691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:30.887196 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:30.887172    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"]
Sep 07 11:33:30.889679 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:33:30.889650    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7566aea8_4484_418c_9887_486f12ac16ec.slice/crio-86d71c36d0395214cf32ba7ada92d4a7085cd56280c4020651dfd04ac1ba5e96 WatchSource:0}: Error finding container 86d71c36d0395214cf32ba7ada92d4a7085cd56280c4020651dfd04ac1ba5e96: Status 404 returned error can't find the container with id 86d71c36d0395214cf32ba7ada92d4a7085cd56280c4020651dfd04ac1ba5e96
Sep 07 11:33:31.570277 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:31.570224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2" event={"ID":"7566aea8-4484-418c-9887-486f12ac16ec","Type":"ContainerStarted","Data":"86d71c36d0395214cf32ba7ada92d4a7085cd56280c4020651dfd04ac1ba5e96"}
Sep 07 11:33:32.576484 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:32.576443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2" event={"ID":"7566aea8-4484-418c-9887-486f12ac16ec","Type":"ContainerStarted","Data":"b024cb16b10b0ab9960f8eb05f7664bcf94dd6cd3777e2a8b50f369bcb8d3900"}
Sep 07 11:33:32.596966 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:32.596919    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2" podStartSLOduration=1.607930864 podStartE2EDuration="2.596904851s" totalImagesPullingTime="988.973987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:33:32.595160046 +0000 UTC m=+662.300940736" watchObservedRunningTime="2026-09-07 11:33:32.596904851 +0000 UTC m=+662.302685565"
Sep 07 11:33:40.613853 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:40.613817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:40.614299 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:40.613880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:40.626141 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:40.626120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:33:41.628807 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:33:41.628766    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8l9r2"
Sep 07 11:34:22.196214 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.196182    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"]
Sep 07 11:34:22.199965 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.199946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.202640 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.202623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 07 11:34:22.210954 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.210931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"]
Sep 07 11:34:22.250644 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.250610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.250836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.250676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbhh\" (UniqueName: \"kubernetes.io/projected/6675f584-648a-4b6d-b281-566783d4beb2-kube-api-access-brbhh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.250836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.250703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.250836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.250719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.250836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.250746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.250836 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.250763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6675f584-648a-4b6d-b281-566783d4beb2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352272 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbhh\" (UniqueName: \"kubernetes.io/projected/6675f584-648a-4b6d-b281-566783d4beb2-kube-api-access-brbhh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352272 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352503 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352503 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352503 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6675f584-648a-4b6d-b281-566783d4beb2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352503 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352710 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352785 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.352879 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.352861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.354701 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.354678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6675f584-648a-4b6d-b281-566783d4beb2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.355074 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.355054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6675f584-648a-4b6d-b281-566783d4beb2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.360588 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.360561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbhh\" (UniqueName: \"kubernetes.io/projected/6675f584-648a-4b6d-b281-566783d4beb2-kube-api-access-brbhh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt\" (UID: \"6675f584-648a-4b6d-b281-566783d4beb2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.512749 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.512714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:22.788960 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.788933    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"]
Sep 07 11:34:22.791130 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:34:22.791100    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6675f584_648a_4b6d_b281_566783d4beb2.slice/crio-bf6bb322e02c7d411c25f896532949c4459dbe7c59c00aed4a1866e36615836b WatchSource:0}: Error finding container bf6bb322e02c7d411c25f896532949c4459dbe7c59c00aed4a1866e36615836b: Status 404 returned error can't find the container with id bf6bb322e02c7d411c25f896532949c4459dbe7c59c00aed4a1866e36615836b
Sep 07 11:34:22.817777 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:22.817752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt" event={"ID":"6675f584-648a-4b6d-b281-566783d4beb2","Type":"ContainerStarted","Data":"bf6bb322e02c7d411c25f896532949c4459dbe7c59c00aed4a1866e36615836b"}
Sep 07 11:34:23.823711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:23.823672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt" event={"ID":"6675f584-648a-4b6d-b281-566783d4beb2","Type":"ContainerStarted","Data":"d8ce0a028551601c0fcc24f411f298ae6830911200926b3ce449bd547b72e3d0"}
Sep 07 11:34:23.847611 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:23.847542    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt" podStartSLOduration=1.6641010440000001 podStartE2EDuration="1.847519724s" totalImagesPullingTime="183.41868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:34:23.845642563 +0000 UTC m=+713.551423254" watchObservedRunningTime="2026-09-07 11:34:23.847519724 +0000 UTC m=+713.553300415"
Sep 07 11:34:32.513693 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:32.513654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:32.514151 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:32.513712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:32.526016 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:32.525990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:32.877210 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:32.877141    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt"
Sep 07 11:34:38.604595 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.604561    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"]
Sep 07 11:34:38.608513 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.608491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.611052 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.611031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 07 11:34:38.619927 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.619904    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"]
Sep 07 11:34:38.692482 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.692449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq487\" (UniqueName: \"kubernetes.io/projected/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-kube-api-access-cq487\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.692637 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.692495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.692637 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.692520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.692637 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.692624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.692746 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.692661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.692746 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.692678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.793882 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.793833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.794047 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.793907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.794047 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.793932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.794047 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.793990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq487\" (UniqueName: \"kubernetes.io/projected/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-kube-api-access-cq487\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.794047 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.794042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.794249 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.794087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.794304 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.794274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.794355 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.794319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.794402 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.794385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.796279 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.796255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.796410 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.796390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.803863 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.803827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq487\" (UniqueName: \"kubernetes.io/projected/237d0dd9-78bd-4d5c-9d49-ffdc30bda092-kube-api-access-cq487\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp\" (UID: \"237d0dd9-78bd-4d5c-9d49-ffdc30bda092\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:38.919196 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:38.919117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:39.201333 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:39.201309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"]
Sep 07 11:34:39.203222 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:34:39.203194    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod237d0dd9_78bd_4d5c_9d49_ffdc30bda092.slice/crio-bf935c1dbe2de090a259385b81508ac50b7120a5ddcb685bb0082015a2fc69aa WatchSource:0}: Error finding container bf935c1dbe2de090a259385b81508ac50b7120a5ddcb685bb0082015a2fc69aa: Status 404 returned error can't find the container with id bf935c1dbe2de090a259385b81508ac50b7120a5ddcb685bb0082015a2fc69aa
Sep 07 11:34:39.898344 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:39.898306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp" event={"ID":"237d0dd9-78bd-4d5c-9d49-ffdc30bda092","Type":"ContainerStarted","Data":"965d6c0dd0975b1314ae2718a5c1375b7d9d9e5e132e13f7905a6f13976a6d0f"}
Sep 07 11:34:39.898725 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:39.898359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp" event={"ID":"237d0dd9-78bd-4d5c-9d49-ffdc30bda092","Type":"ContainerStarted","Data":"bf935c1dbe2de090a259385b81508ac50b7120a5ddcb685bb0082015a2fc69aa"}
Sep 07 11:34:39.917420 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:39.917354    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp" podStartSLOduration=1.717075203 podStartE2EDuration="1.917337758s" totalImagesPullingTime="200.262555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:34:39.91588743 +0000 UTC m=+729.621668123" watchObservedRunningTime="2026-09-07 11:34:39.917337758 +0000 UTC m=+729.623118447"
Sep 07 11:34:48.919290 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:48.919246    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:48.919290 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:48.919303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:48.931774 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:48.931745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:34:48.955972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:34:48.955943    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp"
Sep 07 11:36:05.121712 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.121630    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"]
Sep 07 11:36:05.125110 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.125085    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.127450 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.127431    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b7e4c1-serving-cert\""
Sep 07 11:36:05.135446 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.135419    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"]
Sep 07 11:36:05.226682 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.226644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.226682 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.226686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-metrics-certs\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.227133 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.226725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxg55\" (UniqueName: \"kubernetes.io/projected/72e4af22-f976-4b4f-b162-298fc6f6b440-kube-api-access-xxg55\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.227133 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.226841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e4af22-f976-4b4f-b162-298fc6f6b440-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.327562 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.327527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e4af22-f976-4b4f-b162-298fc6f6b440-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.327731 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.327582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.327731 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.327608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-metrics-certs\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.327731 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.327642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxg55\" (UniqueName: \"kubernetes.io/projected/72e4af22-f976-4b4f-b162-298fc6f6b440-kube-api-access-xxg55\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.327879 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:36:05.327761    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-serving-cert: secret "maas-api-e2e-inf-a-b7e4c1-serving-cert" not found
Sep 07 11:36:05.327879 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:36:05.327860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls podName:72e4af22-f976-4b4f-b162-298fc6f6b440 nodeName:}" failed. No retries permitted until 2026-09-07 11:36:05.827839933 +0000 UTC m=+815.533620615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls") pod "maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd" (UID: "72e4af22-f976-4b4f-b162-298fc6f6b440") : secret "maas-api-e2e-inf-a-b7e4c1-serving-cert" not found
Sep 07 11:36:05.328215 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.328192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e4af22-f976-4b4f-b162-298fc6f6b440-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.330161 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.330142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-metrics-certs\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.336546 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.336500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxg55\" (UniqueName: \"kubernetes.io/projected/72e4af22-f976-4b4f-b162-298fc6f6b440-kube-api-access-xxg55\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.831935 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.831901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:05.834352 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:05.834318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls\") pod \"maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:06.036603 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:06.036560    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:06.317298 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:06.317186    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"]
Sep 07 11:36:06.319527 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:36:06.319498    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e4af22_f976_4b4f_b162_298fc6f6b440.slice/crio-55d3ca8287e351ddde1fab060e326351ad49d260351254ba7970de04e5b38e31 WatchSource:0}: Error finding container 55d3ca8287e351ddde1fab060e326351ad49d260351254ba7970de04e5b38e31: Status 404 returned error can't find the container with id 55d3ca8287e351ddde1fab060e326351ad49d260351254ba7970de04e5b38e31
Sep 07 11:36:07.311192 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:07.311158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd" event={"ID":"72e4af22-f976-4b4f-b162-298fc6f6b440","Type":"ContainerStarted","Data":"6f67138216df9142349bbb82392ee12a586c18def25768f8edad91ffe898f1e4"}
Sep 07 11:36:07.311388 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:07.311209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd" event={"ID":"72e4af22-f976-4b4f-b162-298fc6f6b440","Type":"ContainerStarted","Data":"55d3ca8287e351ddde1fab060e326351ad49d260351254ba7970de04e5b38e31"}
Sep 07 11:36:07.311388 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:07.311307    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:07.362968 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:07.362913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd" podStartSLOduration=1.933075938 podStartE2EDuration="2.362897517s" totalImagesPullingTime="429.821579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:07.361013413 +0000 UTC m=+817.066794102" watchObservedRunningTime="2026-09-07 11:36:07.362897517 +0000 UTC m=+817.068678207"
Sep 07 11:36:13.322245 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.322216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:36:13.714687 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.714649    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"]
Sep 07 11:36:13.718619 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.718596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.723444 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.722690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c30b7add-serving-cert\""
Sep 07 11:36:13.730590 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.730552    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"]
Sep 07 11:36:13.793493 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.793456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6jsv\" (UniqueName: \"kubernetes.io/projected/ffff0fdc-e3eb-4a8c-9704-b14458db5182-kube-api-access-r6jsv\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.793648 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.793501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-metrics-certs\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.793648 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.793561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff0fdc-e3eb-4a8c-9704-b14458db5182-openshift-service-ca\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.793738 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.793647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.894177 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.894137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.894342 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.894194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6jsv\" (UniqueName: \"kubernetes.io/projected/ffff0fdc-e3eb-4a8c-9704-b14458db5182-kube-api-access-r6jsv\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.894342 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.894227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-metrics-certs\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.894342 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.894253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff0fdc-e3eb-4a8c-9704-b14458db5182-openshift-service-ca\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.894342 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:36:13.894285    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-serving-cert: secret "maas-api-e2e-ait-c30b7add-serving-cert" not found
Sep 07 11:36:13.894482 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:36:13.894352    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls podName:ffff0fdc-e3eb-4a8c-9704-b14458db5182 nodeName:}" failed. No retries permitted until 2026-09-07 11:36:14.394332792 +0000 UTC m=+824.100113464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls") pod "maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq" (UID: "ffff0fdc-e3eb-4a8c-9704-b14458db5182") : secret "maas-api-e2e-ait-c30b7add-serving-cert" not found
Sep 07 11:36:13.895056 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.895037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff0fdc-e3eb-4a8c-9704-b14458db5182-openshift-service-ca\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.896705 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.896688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-metrics-certs\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:13.902294 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:13.902277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6jsv\" (UniqueName: \"kubernetes.io/projected/ffff0fdc-e3eb-4a8c-9704-b14458db5182-kube-api-access-r6jsv\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:14.398271 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:14.398238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:14.400866 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:14.400839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls\") pod \"maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:14.634814 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:14.634758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:14.906179 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:14.906024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"]
Sep 07 11:36:14.908663 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:36:14.908636    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffff0fdc_e3eb_4a8c_9704_b14458db5182.slice/crio-652afbbb0cd1ea5431e4b639027271513b0091184b9edaaa53a58487f876c4bb WatchSource:0}: Error finding container 652afbbb0cd1ea5431e4b639027271513b0091184b9edaaa53a58487f876c4bb: Status 404 returned error can't find the container with id 652afbbb0cd1ea5431e4b639027271513b0091184b9edaaa53a58487f876c4bb
Sep 07 11:36:15.357285 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:15.357247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq" event={"ID":"ffff0fdc-e3eb-4a8c-9704-b14458db5182","Type":"ContainerStarted","Data":"652afbbb0cd1ea5431e4b639027271513b0091184b9edaaa53a58487f876c4bb"}
Sep 07 11:36:16.363333 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:16.363300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq" event={"ID":"ffff0fdc-e3eb-4a8c-9704-b14458db5182","Type":"ContainerStarted","Data":"64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67"}
Sep 07 11:36:16.363718 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:16.363410    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:16.383161 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:16.383117    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq" podStartSLOduration=2.9811754 podStartE2EDuration="3.383103686s" totalImagesPullingTime="401.928286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:16.381964105 +0000 UTC m=+826.087744797" watchObservedRunningTime="2026-09-07 11:36:16.383103686 +0000 UTC m=+826.088884376"
Sep 07 11:36:18.108021 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.107986    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"]
Sep 07 11:36:18.112356 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.112333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.115532 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.115510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-152c50-serving-cert\""
Sep 07 11:36:18.125751 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.125724    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"]
Sep 07 11:36:18.131104 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.131070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.131212 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.131125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2459\" (UniqueName: \"kubernetes.io/projected/089ee6d3-53a1-49dd-9e03-9bedf132bd12-kube-api-access-m2459\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.131278 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.131242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089ee6d3-53a1-49dd-9e03-9bedf132bd12-openshift-service-ca\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.131336 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.131292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-metrics-certs\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.232649 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.232613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-metrics-certs\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.232888 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.232678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.232888 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.232723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2459\" (UniqueName: \"kubernetes.io/projected/089ee6d3-53a1-49dd-9e03-9bedf132bd12-kube-api-access-m2459\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.232888 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:36:18.232784    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-152c50-serving-cert: secret "maas-api-e2e-autores-152c50-serving-cert" not found
Sep 07 11:36:18.232888 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.232871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089ee6d3-53a1-49dd-9e03-9bedf132bd12-openshift-service-ca\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.232888 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:36:18.232889    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls podName:089ee6d3-53a1-49dd-9e03-9bedf132bd12 nodeName:}" failed. No retries permitted until 2026-09-07 11:36:18.732866748 +0000 UTC m=+828.438647430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls") pod "maas-api-e2e-autores-152c50-765975f6f5-jw76x" (UID: "089ee6d3-53a1-49dd-9e03-9bedf132bd12") : secret "maas-api-e2e-autores-152c50-serving-cert" not found
Sep 07 11:36:18.233601 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.233576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089ee6d3-53a1-49dd-9e03-9bedf132bd12-openshift-service-ca\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.235065 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.235043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-metrics-certs\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.244597 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.244575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2459\" (UniqueName: \"kubernetes.io/projected/089ee6d3-53a1-49dd-9e03-9bedf132bd12-kube-api-access-m2459\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.737741 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.737703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:18.740659 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:18.740624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls\") pod \"maas-api-e2e-autores-152c50-765975f6f5-jw76x\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:19.026001 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:19.025914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:19.300816 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:19.300779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"]
Sep 07 11:36:19.302784 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:36:19.302756    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089ee6d3_53a1_49dd_9e03_9bedf132bd12.slice/crio-f138a3b43c54a29344b65a04a7a34536f566dfb1f983a89ac32cfcab086bb92a WatchSource:0}: Error finding container f138a3b43c54a29344b65a04a7a34536f566dfb1f983a89ac32cfcab086bb92a: Status 404 returned error can't find the container with id f138a3b43c54a29344b65a04a7a34536f566dfb1f983a89ac32cfcab086bb92a
Sep 07 11:36:19.379051 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:19.379017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x" event={"ID":"089ee6d3-53a1-49dd-9e03-9bedf132bd12","Type":"ContainerStarted","Data":"f138a3b43c54a29344b65a04a7a34536f566dfb1f983a89ac32cfcab086bb92a"}
Sep 07 11:36:20.386043 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:20.386005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x" event={"ID":"089ee6d3-53a1-49dd-9e03-9bedf132bd12","Type":"ContainerStarted","Data":"de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2"}
Sep 07 11:36:20.386418 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:20.386137    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:20.405826 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:20.405762    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x" podStartSLOduration=2.014443312 podStartE2EDuration="2.405745106s" totalImagesPullingTime="391.301794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:20.404202819 +0000 UTC m=+830.109983510" watchObservedRunningTime="2026-09-07 11:36:20.405745106 +0000 UTC m=+830.111525796"
Sep 07 11:36:22.373377 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:22.373348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:26.396733 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:26.396702    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:36:34.218411 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.218379    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"]
Sep 07 11:36:34.222320 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.222294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.225566 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.225535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ef61c5-serving-cert\""
Sep 07 11:36:34.234480 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.234456    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"]
Sep 07 11:36:34.280020 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.279988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-metrics-certs\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.280182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.280038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aebcbe-0d94-4fb8-81f6-ca18269732d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.280182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.280118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7qd\" (UniqueName: \"kubernetes.io/projected/56aebcbe-0d94-4fb8-81f6-ca18269732d7-kube-api-access-tg7qd\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.280182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.280141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.380745 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.380712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aebcbe-0d94-4fb8-81f6-ca18269732d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.380920 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.380830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7qd\" (UniqueName: \"kubernetes.io/projected/56aebcbe-0d94-4fb8-81f6-ca18269732d7-kube-api-access-tg7qd\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.380920 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.380857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.380920 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.380909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-metrics-certs\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.381583 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.381551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aebcbe-0d94-4fb8-81f6-ca18269732d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.383298 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.383276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-metrics-certs\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.383396 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.383368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.388785 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.388764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7qd\" (UniqueName: \"kubernetes.io/projected/56aebcbe-0d94-4fb8-81f6-ca18269732d7-kube-api-access-tg7qd\") pod \"maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.534230 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.534158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:34.821065 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:34.820816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"]
Sep 07 11:36:34.824091 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:36:34.824060    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56aebcbe_0d94_4fb8_81f6_ca18269732d7.slice/crio-f4aa2d80ed1d1c94beb57fe7241909318853e14cca7971ced55e9c41ae9eeeb8 WatchSource:0}: Error finding container f4aa2d80ed1d1c94beb57fe7241909318853e14cca7971ced55e9c41ae9eeeb8: Status 404 returned error can't find the container with id f4aa2d80ed1d1c94beb57fe7241909318853e14cca7971ced55e9c41ae9eeeb8
Sep 07 11:36:35.459624 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:35.459585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl" event={"ID":"56aebcbe-0d94-4fb8-81f6-ca18269732d7","Type":"ContainerStarted","Data":"b6eecbc4e48d7bfd38d3bf819a8ab658cf0e536dfd31dcc4d32b5beda24aaf07"}
Sep 07 11:36:35.460035 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:35.459649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl" event={"ID":"56aebcbe-0d94-4fb8-81f6-ca18269732d7","Type":"ContainerStarted","Data":"f4aa2d80ed1d1c94beb57fe7241909318853e14cca7971ced55e9c41ae9eeeb8"}
Sep 07 11:36:35.460035 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:35.459736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:35.478168 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:35.478124    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl" podStartSLOduration=1.072116826 podStartE2EDuration="1.47810665s" totalImagesPullingTime="405.989824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:36:35.476686205 +0000 UTC m=+845.182466919" watchObservedRunningTime="2026-09-07 11:36:35.47810665 +0000 UTC m=+845.183887339"
Sep 07 11:36:41.473416 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:41.473384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:36:54.477387 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.477348    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"]
Sep 07 11:36:54.477860 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.477646    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq" podUID="ffff0fdc-e3eb-4a8c-9704-b14458db5182" containerName="maas-api" containerID="cri-o://64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67" gracePeriod=30
Sep 07 11:36:54.724549 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.724525    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:54.874638 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.874552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff0fdc-e3eb-4a8c-9704-b14458db5182-openshift-service-ca\") pod \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") "
Sep 07 11:36:54.874638 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.874609    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls\") pod \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") "
Sep 07 11:36:54.874882 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.874639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-metrics-certs\") pod \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") "
Sep 07 11:36:54.874882 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.874679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6jsv\" (UniqueName: \"kubernetes.io/projected/ffff0fdc-e3eb-4a8c-9704-b14458db5182-kube-api-access-r6jsv\") pod \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\" (UID: \"ffff0fdc-e3eb-4a8c-9704-b14458db5182\") "
Sep 07 11:36:54.874982 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.874933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffff0fdc-e3eb-4a8c-9704-b14458db5182-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffff0fdc-e3eb-4a8c-9704-b14458db5182" (UID: "ffff0fdc-e3eb-4a8c-9704-b14458db5182"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:36:54.877287 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.877255    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ffff0fdc-e3eb-4a8c-9704-b14458db5182" (UID: "ffff0fdc-e3eb-4a8c-9704-b14458db5182"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:36:54.877440 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.877281    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffff0fdc-e3eb-4a8c-9704-b14458db5182" (UID: "ffff0fdc-e3eb-4a8c-9704-b14458db5182"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:36:54.877515 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.877476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffff0fdc-e3eb-4a8c-9704-b14458db5182-kube-api-access-r6jsv" (OuterVolumeSpecName: "kube-api-access-r6jsv") pod "ffff0fdc-e3eb-4a8c-9704-b14458db5182" (UID: "ffff0fdc-e3eb-4a8c-9704-b14458db5182"). InnerVolumeSpecName "kube-api-access-r6jsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:36:54.975766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.975728    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff0fdc-e3eb-4a8c-9704-b14458db5182-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:36:54.975766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.975757    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:36:54.975766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.975771    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffff0fdc-e3eb-4a8c-9704-b14458db5182-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:36:54.975985 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:54.975782    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6jsv\" (UniqueName: \"kubernetes.io/projected/ffff0fdc-e3eb-4a8c-9704-b14458db5182-kube-api-access-r6jsv\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:36:55.562495 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.562460    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffff0fdc-e3eb-4a8c-9704-b14458db5182" containerID="64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67" exitCode=0
Sep 07 11:36:55.562962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.562501    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"
Sep 07 11:36:55.562962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.562545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq" event={"ID":"ffff0fdc-e3eb-4a8c-9704-b14458db5182","Type":"ContainerDied","Data":"64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67"}
Sep 07 11:36:55.562962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.562613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq" event={"ID":"ffff0fdc-e3eb-4a8c-9704-b14458db5182","Type":"ContainerDied","Data":"652afbbb0cd1ea5431e4b639027271513b0091184b9edaaa53a58487f876c4bb"}
Sep 07 11:36:55.562962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.562649    2581 scope.go:117] "RemoveContainer" containerID="64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67"
Sep 07 11:36:55.573642 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.572770    2581 scope.go:117] "RemoveContainer" containerID="64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67"
Sep 07 11:36:55.574155 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:36:55.574134    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67\": container with ID starting with 64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67 not found: ID does not exist" containerID="64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67"
Sep 07 11:36:55.574258 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.574162    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67"} err="failed to get container status \"64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67\": rpc error: code = NotFound desc = could not find container \"64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67\": container with ID starting with 64f7f6a0f3df1217cc6e8418488cc1c001f5889a7f0538c016c7c2d26d862b67 not found: ID does not exist"
Sep 07 11:36:55.580642 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.580619    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"]
Sep 07 11:36:55.582532 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:55.582511    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c30b7add-74db4f764b-b4dzq"]
Sep 07 11:36:56.942693 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:56.942645    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffff0fdc-e3eb-4a8c-9704-b14458db5182" path="/var/lib/kubelet/pods/ffff0fdc-e3eb-4a8c-9704-b14458db5182/volumes"
Sep 07 11:36:59.095334 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.095297    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"]
Sep 07 11:36:59.102862 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.102051    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffff0fdc-e3eb-4a8c-9704-b14458db5182" containerName="maas-api"
Sep 07 11:36:59.102862 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.102085    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffff0fdc-e3eb-4a8c-9704-b14458db5182" containerName="maas-api"
Sep 07 11:36:59.102862 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.102338    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffff0fdc-e3eb-4a8c-9704-b14458db5182" containerName="maas-api"
Sep 07 11:36:59.112067 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.112036    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"]
Sep 07 11:36:59.112211 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.112151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.114496 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.114476    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-25494d09-serving-cert\""
Sep 07 11:36:59.212288 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.212079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-maas-api-tls\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.212288 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.212172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfpm\" (UniqueName: \"kubernetes.io/projected/37998bf6-29a9-404c-922a-7b5c96dab156-kube-api-access-8rfpm\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.212288 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.212241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37998bf6-29a9-404c-922a-7b5c96dab156-openshift-service-ca\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.212628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.212526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-metrics-certs\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.313179 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.313150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-maas-api-tls\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.313331 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.313189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfpm\" (UniqueName: \"kubernetes.io/projected/37998bf6-29a9-404c-922a-7b5c96dab156-kube-api-access-8rfpm\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.313331 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.313320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37998bf6-29a9-404c-922a-7b5c96dab156-openshift-service-ca\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.313449 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.313352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-metrics-certs\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.314034 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.314009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37998bf6-29a9-404c-922a-7b5c96dab156-openshift-service-ca\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.315737 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.315712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-maas-api-tls\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.315849 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.315763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-metrics-certs\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.321525 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.321501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfpm\" (UniqueName: \"kubernetes.io/projected/37998bf6-29a9-404c-922a-7b5c96dab156-kube-api-access-8rfpm\") pod \"maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.423669 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.423587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:36:59.702410 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:36:59.702384    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"]
Sep 07 11:36:59.704464 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:36:59.704434    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37998bf6_29a9_404c_922a_7b5c96dab156.slice/crio-d974374925f403163052c2b9910da5586a810f09f2bac20a7047b36e1b1bd17e WatchSource:0}: Error finding container d974374925f403163052c2b9910da5586a810f09f2bac20a7047b36e1b1bd17e: Status 404 returned error can't find the container with id d974374925f403163052c2b9910da5586a810f09f2bac20a7047b36e1b1bd17e
Sep 07 11:37:00.591964 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:00.591924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9" event={"ID":"37998bf6-29a9-404c-922a-7b5c96dab156","Type":"ContainerStarted","Data":"3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52"}
Sep 07 11:37:00.592357 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:00.591978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9" event={"ID":"37998bf6-29a9-404c-922a-7b5c96dab156","Type":"ContainerStarted","Data":"d974374925f403163052c2b9910da5586a810f09f2bac20a7047b36e1b1bd17e"}
Sep 07 11:37:00.592357 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:00.592050    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:37:00.612182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:00.612128    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9" podStartSLOduration=1.237972197 podStartE2EDuration="1.612111096s" totalImagesPullingTime="374.138899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:37:00.610110441 +0000 UTC m=+870.315891134" watchObservedRunningTime="2026-09-07 11:37:00.612111096 +0000 UTC m=+870.317891785"
Sep 07 11:37:06.604178 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:06.604149    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:37:27.290015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.289976    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"]
Sep 07 11:37:27.294461 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.294432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.298835 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.298816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-6580730c-kserve-self-signed-certs\""
Sep 07 11:37:27.313282 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.313246    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"]
Sep 07 11:37:27.375142 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.375101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb64790-73cf-419b-8f73-f021e9d56ff9-tls-certs\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.375312 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.375160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-home\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.375312 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.375189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-model-cache\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.375312 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.375260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-tmp-dir\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.375418 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.375330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-dshm\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.375418 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.375351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z97j\" (UniqueName: \"kubernetes.io/projected/1bb64790-73cf-419b-8f73-f021e9d56ff9-kube-api-access-8z97j\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.476503 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.476466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-tmp-dir\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.476692 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.476560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-dshm\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.476692 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.476584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z97j\" (UniqueName: \"kubernetes.io/projected/1bb64790-73cf-419b-8f73-f021e9d56ff9-kube-api-access-8z97j\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.476692 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.476612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb64790-73cf-419b-8f73-f021e9d56ff9-tls-certs\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.476692 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.476655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-home\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.476692 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.476679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-model-cache\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.476979 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.476946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-tmp-dir\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.477092 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.477065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-model-cache\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.477171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.477104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-home\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.478908 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.478884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-dshm\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.479193 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.479173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb64790-73cf-419b-8f73-f021e9d56ff9-tls-certs\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.486842 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.486816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z97j\" (UniqueName: \"kubernetes.io/projected/1bb64790-73cf-419b-8f73-f021e9d56ff9-kube-api-access-8z97j\") pod \"e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") " pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.603756 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.603660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:27.921174 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:27.921146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"]
Sep 07 11:37:27.922959 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:37:27.922930    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bb64790_73cf_419b_8f73_f021e9d56ff9.slice/crio-f13d8b6eba44225a2b69015141bbdb2a3f3bd6054d974f93fafed4005d0e609c WatchSource:0}: Error finding container f13d8b6eba44225a2b69015141bbdb2a3f3bd6054d974f93fafed4005d0e609c: Status 404 returned error can't find the container with id f13d8b6eba44225a2b69015141bbdb2a3f3bd6054d974f93fafed4005d0e609c
Sep 07 11:37:28.742356 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:28.742318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj" event={"ID":"1bb64790-73cf-419b-8f73-f021e9d56ff9","Type":"ContainerStarted","Data":"e117ec58f5f86b9aa98cf9a0ad9231f9455ee9f57aec45b95aad0401c04b6dc5"}
Sep 07 11:37:28.742750 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:28.742367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj" event={"ID":"1bb64790-73cf-419b-8f73-f021e9d56ff9","Type":"ContainerStarted","Data":"f13d8b6eba44225a2b69015141bbdb2a3f3bd6054d974f93fafed4005d0e609c"}
Sep 07 11:37:30.775149 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:30.775092    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj" podStartSLOduration=3.77507146 podStartE2EDuration="3.77507146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:37:28.777318381 +0000 UTC m=+898.483099070" watchObservedRunningTime="2026-09-07 11:37:30.77507146 +0000 UTC m=+900.480852152"
Sep 07 11:37:30.775727 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:30.775655    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"]
Sep 07 11:37:30.775929 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:30.775904    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9" podUID="37998bf6-29a9-404c-922a-7b5c96dab156" containerName="maas-api" containerID="cri-o://3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52" gracePeriod=30
Sep 07 11:37:31.037693 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.037629    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:37:31.212328 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.212285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rfpm\" (UniqueName: \"kubernetes.io/projected/37998bf6-29a9-404c-922a-7b5c96dab156-kube-api-access-8rfpm\") pod \"37998bf6-29a9-404c-922a-7b5c96dab156\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") "
Sep 07 11:37:31.212519 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.212346    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-maas-api-tls\") pod \"37998bf6-29a9-404c-922a-7b5c96dab156\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") "
Sep 07 11:37:31.212519 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.212384    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-metrics-certs\") pod \"37998bf6-29a9-404c-922a-7b5c96dab156\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") "
Sep 07 11:37:31.212519 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.212436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37998bf6-29a9-404c-922a-7b5c96dab156-openshift-service-ca\") pod \"37998bf6-29a9-404c-922a-7b5c96dab156\" (UID: \"37998bf6-29a9-404c-922a-7b5c96dab156\") "
Sep 07 11:37:31.212908 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.212879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37998bf6-29a9-404c-922a-7b5c96dab156-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37998bf6-29a9-404c-922a-7b5c96dab156" (UID: "37998bf6-29a9-404c-922a-7b5c96dab156"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:37:31.214633 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.214599    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37998bf6-29a9-404c-922a-7b5c96dab156-kube-api-access-8rfpm" (OuterVolumeSpecName: "kube-api-access-8rfpm") pod "37998bf6-29a9-404c-922a-7b5c96dab156" (UID: "37998bf6-29a9-404c-922a-7b5c96dab156"). InnerVolumeSpecName "kube-api-access-8rfpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:37:31.214633 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.214608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "37998bf6-29a9-404c-922a-7b5c96dab156" (UID: "37998bf6-29a9-404c-922a-7b5c96dab156"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:37:31.214763 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.214692    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37998bf6-29a9-404c-922a-7b5c96dab156" (UID: "37998bf6-29a9-404c-922a-7b5c96dab156"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:37:31.314090 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.314003    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:37:31.314090 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.314036    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37998bf6-29a9-404c-922a-7b5c96dab156-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:37:31.314090 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.314046    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rfpm\" (UniqueName: \"kubernetes.io/projected/37998bf6-29a9-404c-922a-7b5c96dab156-kube-api-access-8rfpm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:37:31.314090 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.314056    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37998bf6-29a9-404c-922a-7b5c96dab156-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:37:31.762404 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.762369    2581 generic.go:356] "Generic (PLEG): container finished" podID="37998bf6-29a9-404c-922a-7b5c96dab156" containerID="3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52" exitCode=0
Sep 07 11:37:31.762609 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.762418    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"
Sep 07 11:37:31.762609 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.762458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9" event={"ID":"37998bf6-29a9-404c-922a-7b5c96dab156","Type":"ContainerDied","Data":"3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52"}
Sep 07 11:37:31.762609 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.762514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9" event={"ID":"37998bf6-29a9-404c-922a-7b5c96dab156","Type":"ContainerDied","Data":"d974374925f403163052c2b9910da5586a810f09f2bac20a7047b36e1b1bd17e"}
Sep 07 11:37:31.762609 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.762545    2581 scope.go:117] "RemoveContainer" containerID="3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52"
Sep 07 11:37:31.772028 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.772010    2581 scope.go:117] "RemoveContainer" containerID="3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52"
Sep 07 11:37:31.772318 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:37:31.772298    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52\": container with ID starting with 3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52 not found: ID does not exist" containerID="3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52"
Sep 07 11:37:31.772363 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.772328    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52"} err="failed to get container status \"3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52\": rpc error: code = NotFound desc = could not find container \"3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52\": container with ID starting with 3ed97951108f52fa76c37b5fe99b6834b97c5b493d658377a2bcee49d22b3e52 not found: ID does not exist"
Sep 07 11:37:31.787801 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.787768    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"]
Sep 07 11:37:31.791379 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:31.791359    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25494d09-5cc444648d-6vbs9"]
Sep 07 11:37:32.945047 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:32.945010    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37998bf6-29a9-404c-922a-7b5c96dab156" path="/var/lib/kubelet/pods/37998bf6-29a9-404c-922a-7b5c96dab156/volumes"
Sep 07 11:37:36.070465 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.070425    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"]
Sep 07 11:37:36.071642 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.071611    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37998bf6-29a9-404c-922a-7b5c96dab156" containerName="maas-api"
Sep 07 11:37:36.071642 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.071644    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="37998bf6-29a9-404c-922a-7b5c96dab156" containerName="maas-api"
Sep 07 11:37:36.071777 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.071750    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="37998bf6-29a9-404c-922a-7b5c96dab156" containerName="maas-api"
Sep 07 11:37:36.077379 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.077350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.080054 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.080034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5e86f370-serving-cert\""
Sep 07 11:37:36.087393 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.087149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"]
Sep 07 11:37:36.261551 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.261511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6z5w\" (UniqueName: \"kubernetes.io/projected/481e2ae2-b596-4785-9dca-062d3fb0e926-kube-api-access-d6z5w\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.262054 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.261637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-metrics-certs\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.262054 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.261680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e2ae2-b596-4785-9dca-062d3fb0e926-openshift-service-ca\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.262054 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.261729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-maas-api-tls\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.363167 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.363076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-metrics-certs\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.363167 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.363134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e2ae2-b596-4785-9dca-062d3fb0e926-openshift-service-ca\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.363371 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.363192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-maas-api-tls\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.363371 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.363237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6z5w\" (UniqueName: \"kubernetes.io/projected/481e2ae2-b596-4785-9dca-062d3fb0e926-kube-api-access-d6z5w\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.363908 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.363883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e2ae2-b596-4785-9dca-062d3fb0e926-openshift-service-ca\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.365665 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.365644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-maas-api-tls\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.365753 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.365696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-metrics-certs\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.375087 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.375063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6z5w\" (UniqueName: \"kubernetes.io/projected/481e2ae2-b596-4785-9dca-062d3fb0e926-kube-api-access-d6z5w\") pod \"maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.390716 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.390692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:36.673374 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.673309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"]
Sep 07 11:37:36.675672 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:37:36.675634    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481e2ae2_b596_4785_9dca_062d3fb0e926.slice/crio-a590338bd78ca76c268a6bbe9f6f0ea64b7d5574a7c463ba292c8c76f64f8bf5 WatchSource:0}: Error finding container a590338bd78ca76c268a6bbe9f6f0ea64b7d5574a7c463ba292c8c76f64f8bf5: Status 404 returned error can't find the container with id a590338bd78ca76c268a6bbe9f6f0ea64b7d5574a7c463ba292c8c76f64f8bf5
Sep 07 11:37:36.789722 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:36.789683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx" event={"ID":"481e2ae2-b596-4785-9dca-062d3fb0e926","Type":"ContainerStarted","Data":"a590338bd78ca76c268a6bbe9f6f0ea64b7d5574a7c463ba292c8c76f64f8bf5"}
Sep 07 11:37:37.604553 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:37.604510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:37.604553 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:37.604565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:37.617382 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:37.617349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:37.796585 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:37.796533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx" event={"ID":"481e2ae2-b596-4785-9dca-062d3fb0e926","Type":"ContainerStarted","Data":"3a499f41c47b93dbbfbf1a4a75b73ce5fde63a62f30fce9bf29677fa50f343e0"}
Sep 07 11:37:37.796827 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:37.796781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:37:37.807919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:37.807892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:37:37.815585 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:37.815535    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx" podStartSLOduration=1.307034251 podStartE2EDuration="1.815519173s" totalImagesPullingTime="508.484922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:37:37.813474578 +0000 UTC m=+907.519255273" watchObservedRunningTime="2026-09-07 11:37:37.815519173 +0000 UTC m=+907.521299862"
Sep 07 11:37:43.807135 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:37:43.807105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:38:06.721067 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:06.721031    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"]
Sep 07 11:38:06.721594 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:06.721278    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx" podUID="481e2ae2-b596-4785-9dca-062d3fb0e926" containerName="maas-api" containerID="cri-o://3a499f41c47b93dbbfbf1a4a75b73ce5fde63a62f30fce9bf29677fa50f343e0" gracePeriod=30
Sep 07 11:38:06.945848 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:06.945784    2581 generic.go:356] "Generic (PLEG): container finished" podID="481e2ae2-b596-4785-9dca-062d3fb0e926" containerID="3a499f41c47b93dbbfbf1a4a75b73ce5fde63a62f30fce9bf29677fa50f343e0" exitCode=0
Sep 07 11:38:06.946024 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:06.945821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx" event={"ID":"481e2ae2-b596-4785-9dca-062d3fb0e926","Type":"ContainerDied","Data":"3a499f41c47b93dbbfbf1a4a75b73ce5fde63a62f30fce9bf29677fa50f343e0"}
Sep 07 11:38:06.980375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:06.980318    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:38:07.056628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.056592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6z5w\" (UniqueName: \"kubernetes.io/projected/481e2ae2-b596-4785-9dca-062d3fb0e926-kube-api-access-d6z5w\") pod \"481e2ae2-b596-4785-9dca-062d3fb0e926\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") "
Sep 07 11:38:07.056821 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.056640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e2ae2-b596-4785-9dca-062d3fb0e926-openshift-service-ca\") pod \"481e2ae2-b596-4785-9dca-062d3fb0e926\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") "
Sep 07 11:38:07.056821 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.056682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-metrics-certs\") pod \"481e2ae2-b596-4785-9dca-062d3fb0e926\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") "
Sep 07 11:38:07.056821 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.056721    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-maas-api-tls\") pod \"481e2ae2-b596-4785-9dca-062d3fb0e926\" (UID: \"481e2ae2-b596-4785-9dca-062d3fb0e926\") "
Sep 07 11:38:07.057074 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.057039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481e2ae2-b596-4785-9dca-062d3fb0e926-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "481e2ae2-b596-4785-9dca-062d3fb0e926" (UID: "481e2ae2-b596-4785-9dca-062d3fb0e926"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:07.058882 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.058854    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "481e2ae2-b596-4785-9dca-062d3fb0e926" (UID: "481e2ae2-b596-4785-9dca-062d3fb0e926"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:38:07.058963 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.058919    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "481e2ae2-b596-4785-9dca-062d3fb0e926" (UID: "481e2ae2-b596-4785-9dca-062d3fb0e926"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:38:07.058963 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.058944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481e2ae2-b596-4785-9dca-062d3fb0e926-kube-api-access-d6z5w" (OuterVolumeSpecName: "kube-api-access-d6z5w") pod "481e2ae2-b596-4785-9dca-062d3fb0e926" (UID: "481e2ae2-b596-4785-9dca-062d3fb0e926"). InnerVolumeSpecName "kube-api-access-d6z5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:07.158388 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.158348    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6z5w\" (UniqueName: \"kubernetes.io/projected/481e2ae2-b596-4785-9dca-062d3fb0e926-kube-api-access-d6z5w\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:07.158388 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.158380    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e2ae2-b596-4785-9dca-062d3fb0e926-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:07.158388 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.158390    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:07.158388 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.158399    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481e2ae2-b596-4785-9dca-062d3fb0e926-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:07.952424 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.952395    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"
Sep 07 11:38:07.952867 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.952440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx" event={"ID":"481e2ae2-b596-4785-9dca-062d3fb0e926","Type":"ContainerDied","Data":"a590338bd78ca76c268a6bbe9f6f0ea64b7d5574a7c463ba292c8c76f64f8bf5"}
Sep 07 11:38:07.952867 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.952504    2581 scope.go:117] "RemoveContainer" containerID="3a499f41c47b93dbbfbf1a4a75b73ce5fde63a62f30fce9bf29677fa50f343e0"
Sep 07 11:38:07.980298 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.980262    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"]
Sep 07 11:38:07.985964 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:07.985935    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e86f370-6d56f995cd-twlzx"]
Sep 07 11:38:08.942539 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:08.942505    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481e2ae2-b596-4785-9dca-062d3fb0e926" path="/var/lib/kubelet/pods/481e2ae2-b596-4785-9dca-062d3fb0e926/volumes"
Sep 07 11:38:12.392834 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.392777    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"]
Sep 07 11:38:12.393412 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.393389    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481e2ae2-b596-4785-9dca-062d3fb0e926" containerName="maas-api"
Sep 07 11:38:12.393484 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.393418    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="481e2ae2-b596-4785-9dca-062d3fb0e926" containerName="maas-api"
Sep 07 11:38:12.393551 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.393537    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="481e2ae2-b596-4785-9dca-062d3fb0e926" containerName="maas-api"
Sep 07 11:38:12.397082 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.397063    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.400159 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.400000    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3f8729d3-serving-cert\""
Sep 07 11:38:12.408847 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.408822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"]
Sep 07 11:38:12.514031 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.514000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-metrics-certs\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.514212 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.514112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8ks\" (UniqueName: \"kubernetes.io/projected/b2224c75-7942-4aa0-9f01-8022de1ea068-kube-api-access-jw8ks\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.514212 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.514170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-maas-api-tls\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.514307 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.514224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224c75-7942-4aa0-9f01-8022de1ea068-openshift-service-ca\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.615547 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.615508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224c75-7942-4aa0-9f01-8022de1ea068-openshift-service-ca\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.615736 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.615577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-metrics-certs\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.615736 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.615644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8ks\" (UniqueName: \"kubernetes.io/projected/b2224c75-7942-4aa0-9f01-8022de1ea068-kube-api-access-jw8ks\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.615736 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.615692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-maas-api-tls\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.616420 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.616393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224c75-7942-4aa0-9f01-8022de1ea068-openshift-service-ca\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.618169 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.618148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-metrics-certs\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.618248 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.618190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-maas-api-tls\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.627488 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.627466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8ks\" (UniqueName: \"kubernetes.io/projected/b2224c75-7942-4aa0-9f01-8022de1ea068-kube-api-access-jw8ks\") pod \"maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.711221 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.711188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:12.995102 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.995076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"]
Sep 07 11:38:12.996866 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:38:12.996843    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2224c75_7942_4aa0_9f01_8022de1ea068.slice/crio-efcd56978f578fbe14b159f70b6de62348361c3e9a2afcc484942416cfe30716 WatchSource:0}: Error finding container efcd56978f578fbe14b159f70b6de62348361c3e9a2afcc484942416cfe30716: Status 404 returned error can't find the container with id efcd56978f578fbe14b159f70b6de62348361c3e9a2afcc484942416cfe30716
Sep 07 11:38:12.998207 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:12.998189    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:38:13.984268 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:13.984234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n" event={"ID":"b2224c75-7942-4aa0-9f01-8022de1ea068","Type":"ContainerStarted","Data":"82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41"}
Sep 07 11:38:13.984666 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:13.984290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n" event={"ID":"b2224c75-7942-4aa0-9f01-8022de1ea068","Type":"ContainerStarted","Data":"efcd56978f578fbe14b159f70b6de62348361c3e9a2afcc484942416cfe30716"}
Sep 07 11:38:13.984666 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:13.984329    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:14.002582 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:14.002532    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n" podStartSLOduration=1.5898375279999999 podStartE2EDuration="2.002515392s" totalImagesPullingTime="412.677864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:14.000135731 +0000 UTC m=+943.705916441" watchObservedRunningTime="2026-09-07 11:38:14.002515392 +0000 UTC m=+943.708296082"
Sep 07 11:38:19.995101 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:19.995067    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:32.913092 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:32.913059    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"]
Sep 07 11:38:44.276605 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.276569    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"]
Sep 07 11:38:44.277157 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.276782    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x" podUID="089ee6d3-53a1-49dd-9e03-9bedf132bd12" containerName="maas-api" containerID="cri-o://de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2" gracePeriod=30
Sep 07 11:38:44.527558 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.527497    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:38:44.623286 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.623249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2459\" (UniqueName: \"kubernetes.io/projected/089ee6d3-53a1-49dd-9e03-9bedf132bd12-kube-api-access-m2459\") pod \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") "
Sep 07 11:38:44.623286 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.623289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls\") pod \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") "
Sep 07 11:38:44.623543 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.623420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-metrics-certs\") pod \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") "
Sep 07 11:38:44.623543 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.623488    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089ee6d3-53a1-49dd-9e03-9bedf132bd12-openshift-service-ca\") pod \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\" (UID: \"089ee6d3-53a1-49dd-9e03-9bedf132bd12\") "
Sep 07 11:38:44.623973 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.623939    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089ee6d3-53a1-49dd-9e03-9bedf132bd12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "089ee6d3-53a1-49dd-9e03-9bedf132bd12" (UID: "089ee6d3-53a1-49dd-9e03-9bedf132bd12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:44.625522 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.625488    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "089ee6d3-53a1-49dd-9e03-9bedf132bd12" (UID: "089ee6d3-53a1-49dd-9e03-9bedf132bd12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:38:44.625637 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.625553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089ee6d3-53a1-49dd-9e03-9bedf132bd12-kube-api-access-m2459" (OuterVolumeSpecName: "kube-api-access-m2459") pod "089ee6d3-53a1-49dd-9e03-9bedf132bd12" (UID: "089ee6d3-53a1-49dd-9e03-9bedf132bd12"). InnerVolumeSpecName "kube-api-access-m2459". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:44.625637 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.625552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "089ee6d3-53a1-49dd-9e03-9bedf132bd12" (UID: "089ee6d3-53a1-49dd-9e03-9bedf132bd12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:38:44.724971 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.724930    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089ee6d3-53a1-49dd-9e03-9bedf132bd12-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:44.724971 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.724961    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2459\" (UniqueName: \"kubernetes.io/projected/089ee6d3-53a1-49dd-9e03-9bedf132bd12-kube-api-access-m2459\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:44.724971 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.724974    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:44.725206 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:44.724987    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089ee6d3-53a1-49dd-9e03-9bedf132bd12-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:45.136210 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.136175    2581 generic.go:356] "Generic (PLEG): container finished" podID="089ee6d3-53a1-49dd-9e03-9bedf132bd12" containerID="de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2" exitCode=0
Sep 07 11:38:45.136403 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.136224    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"
Sep 07 11:38:45.136403 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.136261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x" event={"ID":"089ee6d3-53a1-49dd-9e03-9bedf132bd12","Type":"ContainerDied","Data":"de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2"}
Sep 07 11:38:45.136403 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.136319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x" event={"ID":"089ee6d3-53a1-49dd-9e03-9bedf132bd12","Type":"ContainerDied","Data":"f138a3b43c54a29344b65a04a7a34536f566dfb1f983a89ac32cfcab086bb92a"}
Sep 07 11:38:45.136403 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.136349    2581 scope.go:117] "RemoveContainer" containerID="de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2"
Sep 07 11:38:45.145855 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.145833    2581 scope.go:117] "RemoveContainer" containerID="de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2"
Sep 07 11:38:45.146167 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:38:45.146137    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2\": container with ID starting with de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2 not found: ID does not exist" containerID="de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2"
Sep 07 11:38:45.146233 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.146176    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2"} err="failed to get container status \"de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2\": rpc error: code = NotFound desc = could not find container \"de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2\": container with ID starting with de2b8ca4d3c24ad5f64b9954530c7e9641dad4994e16cf2f75a6a6383962a8b2 not found: ID does not exist"
Sep 07 11:38:45.153901 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.153879    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"]
Sep 07 11:38:45.159847 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:45.159824    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-152c50-765975f6f5-jw76x"]
Sep 07 11:38:46.942716 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:46.942679    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="089ee6d3-53a1-49dd-9e03-9bedf132bd12" path="/var/lib/kubelet/pods/089ee6d3-53a1-49dd-9e03-9bedf132bd12/volumes"
Sep 07 11:38:47.935517 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:47.935473    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj" podUID="1bb64790-73cf-419b-8f73-f021e9d56ff9" containerName="main" containerID="cri-o://e117ec58f5f86b9aa98cf9a0ad9231f9455ee9f57aec45b95aad0401c04b6dc5" gracePeriod=45
Sep 07 11:38:48.155498 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.155465    2581 generic.go:356] "Generic (PLEG): container finished" podID="1bb64790-73cf-419b-8f73-f021e9d56ff9" containerID="e117ec58f5f86b9aa98cf9a0ad9231f9455ee9f57aec45b95aad0401c04b6dc5" exitCode=0
Sep 07 11:38:48.155864 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.155549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj" event={"ID":"1bb64790-73cf-419b-8f73-f021e9d56ff9","Type":"ContainerDied","Data":"e117ec58f5f86b9aa98cf9a0ad9231f9455ee9f57aec45b95aad0401c04b6dc5"}
Sep 07 11:38:48.189081 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.189029    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:38:48.360780 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.360746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-home\") pod \"1bb64790-73cf-419b-8f73-f021e9d56ff9\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") "
Sep 07 11:38:48.360780 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.360784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-tmp-dir\") pod \"1bb64790-73cf-419b-8f73-f021e9d56ff9\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") "
Sep 07 11:38:48.361041 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.360867    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-model-cache\") pod \"1bb64790-73cf-419b-8f73-f021e9d56ff9\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") "
Sep 07 11:38:48.361041 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.360898    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-dshm\") pod \"1bb64790-73cf-419b-8f73-f021e9d56ff9\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") "
Sep 07 11:38:48.361041 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.360941    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb64790-73cf-419b-8f73-f021e9d56ff9-tls-certs\") pod \"1bb64790-73cf-419b-8f73-f021e9d56ff9\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") "
Sep 07 11:38:48.361041 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.361009    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z97j\" (UniqueName: \"kubernetes.io/projected/1bb64790-73cf-419b-8f73-f021e9d56ff9-kube-api-access-8z97j\") pod \"1bb64790-73cf-419b-8f73-f021e9d56ff9\" (UID: \"1bb64790-73cf-419b-8f73-f021e9d56ff9\") "
Sep 07 11:38:48.361242 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.361049    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-home" (OuterVolumeSpecName: "home") pod "1bb64790-73cf-419b-8f73-f021e9d56ff9" (UID: "1bb64790-73cf-419b-8f73-f021e9d56ff9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:48.361242 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.361122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bb64790-73cf-419b-8f73-f021e9d56ff9" (UID: "1bb64790-73cf-419b-8f73-f021e9d56ff9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:48.361242 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.361138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bb64790-73cf-419b-8f73-f021e9d56ff9" (UID: "1bb64790-73cf-419b-8f73-f021e9d56ff9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:48.361406 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.361389    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-home\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.361471 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.361444    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-tmp-dir\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.361471 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.361461    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-model-cache\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.363182 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.363160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb64790-73cf-419b-8f73-f021e9d56ff9-kube-api-access-8z97j" (OuterVolumeSpecName: "kube-api-access-8z97j") pod "1bb64790-73cf-419b-8f73-f021e9d56ff9" (UID: "1bb64790-73cf-419b-8f73-f021e9d56ff9"). InnerVolumeSpecName "kube-api-access-8z97j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:48.363256 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.363200    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb64790-73cf-419b-8f73-f021e9d56ff9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bb64790-73cf-419b-8f73-f021e9d56ff9" (UID: "1bb64790-73cf-419b-8f73-f021e9d56ff9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:38:48.363256 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.363231    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-dshm" (OuterVolumeSpecName: "dshm") pod "1bb64790-73cf-419b-8f73-f021e9d56ff9" (UID: "1bb64790-73cf-419b-8f73-f021e9d56ff9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:38:48.462275 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.462246    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb64790-73cf-419b-8f73-f021e9d56ff9-dshm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.462275 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.462273    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb64790-73cf-419b-8f73-f021e9d56ff9-tls-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:48.462450 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:48.462284    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z97j\" (UniqueName: \"kubernetes.io/projected/1bb64790-73cf-419b-8f73-f021e9d56ff9-kube-api-access-8z97j\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:49.161936 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:49.161909    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"
Sep 07 11:38:49.161936 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:49.161920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj" event={"ID":"1bb64790-73cf-419b-8f73-f021e9d56ff9","Type":"ContainerDied","Data":"f13d8b6eba44225a2b69015141bbdb2a3f3bd6054d974f93fafed4005d0e609c"}
Sep 07 11:38:49.162392 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:49.161975    2581 scope.go:117] "RemoveContainer" containerID="e117ec58f5f86b9aa98cf9a0ad9231f9455ee9f57aec45b95aad0401c04b6dc5"
Sep 07 11:38:49.177972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:49.177951    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"]
Sep 07 11:38:49.182254 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:49.182233    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-6580730c-kserve-5fcd6b999c-bg9fj"]
Sep 07 11:38:50.948670 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:50.948626    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bb64790-73cf-419b-8f73-f021e9d56ff9" path="/var/lib/kubelet/pods/1bb64790-73cf-419b-8f73-f021e9d56ff9/volumes"
Sep 07 11:38:53.283861 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.283755    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"]
Sep 07 11:38:53.284300 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.284061    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n" podUID="b2224c75-7942-4aa0-9f01-8022de1ea068" containerName="maas-api" containerID="cri-o://82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41" gracePeriod=30
Sep 07 11:38:53.527112 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.527088    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:53.612980 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.612900    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-metrics-certs\") pod \"b2224c75-7942-4aa0-9f01-8022de1ea068\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") "
Sep 07 11:38:53.612980 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.612933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-maas-api-tls\") pod \"b2224c75-7942-4aa0-9f01-8022de1ea068\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") "
Sep 07 11:38:53.612980 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.612981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224c75-7942-4aa0-9f01-8022de1ea068-openshift-service-ca\") pod \"b2224c75-7942-4aa0-9f01-8022de1ea068\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") "
Sep 07 11:38:53.613185 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.613022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw8ks\" (UniqueName: \"kubernetes.io/projected/b2224c75-7942-4aa0-9f01-8022de1ea068-kube-api-access-jw8ks\") pod \"b2224c75-7942-4aa0-9f01-8022de1ea068\" (UID: \"b2224c75-7942-4aa0-9f01-8022de1ea068\") "
Sep 07 11:38:53.613393 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.613367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2224c75-7942-4aa0-9f01-8022de1ea068-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2224c75-7942-4aa0-9f01-8022de1ea068" (UID: "b2224c75-7942-4aa0-9f01-8022de1ea068"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:38:53.615148 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.615119    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b2224c75-7942-4aa0-9f01-8022de1ea068" (UID: "b2224c75-7942-4aa0-9f01-8022de1ea068"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:38:53.615268 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.615200    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2224c75-7942-4aa0-9f01-8022de1ea068" (UID: "b2224c75-7942-4aa0-9f01-8022de1ea068"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:38:53.615268 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.615202    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2224c75-7942-4aa0-9f01-8022de1ea068-kube-api-access-jw8ks" (OuterVolumeSpecName: "kube-api-access-jw8ks") pod "b2224c75-7942-4aa0-9f01-8022de1ea068" (UID: "b2224c75-7942-4aa0-9f01-8022de1ea068"). InnerVolumeSpecName "kube-api-access-jw8ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:38:53.713870 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.713839    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8ks\" (UniqueName: \"kubernetes.io/projected/b2224c75-7942-4aa0-9f01-8022de1ea068-kube-api-access-jw8ks\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:53.713870 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.713865    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:53.713870 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.713874    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2224c75-7942-4aa0-9f01-8022de1ea068-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:53.714078 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:53.713882    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224c75-7942-4aa0-9f01-8022de1ea068-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:38:54.189048 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.189013    2581 generic.go:356] "Generic (PLEG): container finished" podID="b2224c75-7942-4aa0-9f01-8022de1ea068" containerID="82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41" exitCode=0
Sep 07 11:38:54.189217 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.189061    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"
Sep 07 11:38:54.189217 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.189109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n" event={"ID":"b2224c75-7942-4aa0-9f01-8022de1ea068","Type":"ContainerDied","Data":"82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41"}
Sep 07 11:38:54.189217 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.189173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n" event={"ID":"b2224c75-7942-4aa0-9f01-8022de1ea068","Type":"ContainerDied","Data":"efcd56978f578fbe14b159f70b6de62348361c3e9a2afcc484942416cfe30716"}
Sep 07 11:38:54.189217 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.189196    2581 scope.go:117] "RemoveContainer" containerID="82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41"
Sep 07 11:38:54.198434 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.198415    2581 scope.go:117] "RemoveContainer" containerID="82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41"
Sep 07 11:38:54.198687 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:38:54.198665    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41\": container with ID starting with 82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41 not found: ID does not exist" containerID="82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41"
Sep 07 11:38:54.198739 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.198695    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41"} err="failed to get container status \"82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41\": rpc error: code = NotFound desc = could not find container \"82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41\": container with ID starting with 82e0d4105de793708b83c0471cb99616057075dc712d36abb04e123adf5f1a41 not found: ID does not exist"
Sep 07 11:38:54.210597 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.210573    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"]
Sep 07 11:38:54.215122 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.215103    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f8729d3-664d6d5dbf-5fg5n"]
Sep 07 11:38:54.942908 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:54.942873    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2224c75-7942-4aa0-9f01-8022de1ea068" path="/var/lib/kubelet/pods/b2224c75-7942-4aa0-9f01-8022de1ea068/volumes"
Sep 07 11:38:56.237766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.237704    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"]
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238355    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb64790-73cf-419b-8f73-f021e9d56ff9" containerName="main"
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238385    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb64790-73cf-419b-8f73-f021e9d56ff9" containerName="main"
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238405    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089ee6d3-53a1-49dd-9e03-9bedf132bd12" containerName="maas-api"
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238414    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="089ee6d3-53a1-49dd-9e03-9bedf132bd12" containerName="maas-api"
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238425    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2224c75-7942-4aa0-9f01-8022de1ea068" containerName="maas-api"
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238433    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2224c75-7942-4aa0-9f01-8022de1ea068" containerName="maas-api"
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238540    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="089ee6d3-53a1-49dd-9e03-9bedf132bd12" containerName="maas-api"
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238554    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2224c75-7942-4aa0-9f01-8022de1ea068" containerName="maas-api"
Sep 07 11:38:56.238850 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.238565    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb64790-73cf-419b-8f73-f021e9d56ff9" containerName="main"
Sep 07 11:38:56.243740 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.243721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.246591 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.246568    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-bf198c-serving-cert\""
Sep 07 11:38:56.251873 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.251846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"]
Sep 07 11:38:56.338840 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.338803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn4w9\" (UniqueName: \"kubernetes.io/projected/a967ae83-c6aa-4939-a1ca-dc93cce4349a-kube-api-access-gn4w9\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.339037 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.338893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-maas-api-tls\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.339037 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.338920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a967ae83-c6aa-4939-a1ca-dc93cce4349a-openshift-service-ca\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.339037 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.338946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-metrics-certs\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.440154 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.440114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4w9\" (UniqueName: \"kubernetes.io/projected/a967ae83-c6aa-4939-a1ca-dc93cce4349a-kube-api-access-gn4w9\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.440341 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.440191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-maas-api-tls\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.440341 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.440211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a967ae83-c6aa-4939-a1ca-dc93cce4349a-openshift-service-ca\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.440341 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.440230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-metrics-certs\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.441051 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.441029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a967ae83-c6aa-4939-a1ca-dc93cce4349a-openshift-service-ca\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.443347 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.443322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-maas-api-tls\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.443463 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.443328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-metrics-certs\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.448779 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.448755    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4w9\" (UniqueName: \"kubernetes.io/projected/a967ae83-c6aa-4939-a1ca-dc93cce4349a-kube-api-access-gn4w9\") pod \"maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.556263 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.556189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:56.829916 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:56.829884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"]
Sep 07 11:38:56.831907 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:38:56.831879    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda967ae83_c6aa_4939_a1ca_dc93cce4349a.slice/crio-ab2dc168a0df802ff744b157a0d948eaeda83ac2acf90077fdde67ff2f5f99a1 WatchSource:0}: Error finding container ab2dc168a0df802ff744b157a0d948eaeda83ac2acf90077fdde67ff2f5f99a1: Status 404 returned error can't find the container with id ab2dc168a0df802ff744b157a0d948eaeda83ac2acf90077fdde67ff2f5f99a1
Sep 07 11:38:57.205773 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:57.205688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h" event={"ID":"a967ae83-c6aa-4939-a1ca-dc93cce4349a","Type":"ContainerStarted","Data":"ab2dc168a0df802ff744b157a0d948eaeda83ac2acf90077fdde67ff2f5f99a1"}
Sep 07 11:38:58.212336 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:58.212303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h" event={"ID":"a967ae83-c6aa-4939-a1ca-dc93cce4349a","Type":"ContainerStarted","Data":"78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237"}
Sep 07 11:38:58.212757 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:58.212362    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:38:58.230896 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:38:58.230852    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h" podStartSLOduration=1.785152145 podStartE2EDuration="2.230834604s" totalImagesPullingTime="445.682459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:58.230210208 +0000 UTC m=+987.935990899" watchObservedRunningTime="2026-09-07 11:38:58.230834604 +0000 UTC m=+987.936615293"
Sep 07 11:39:04.222153 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:39:04.222119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:40:10.297459 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.297416    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"]
Sep 07 11:40:10.571082 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.570991    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h" podUID="a967ae83-c6aa-4939-a1ca-dc93cce4349a" containerName="maas-api" containerID="cri-o://78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237" gracePeriod=30
Sep 07 11:40:10.819126 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.819102    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:40:10.904198 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.904116    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-maas-api-tls\") pod \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") "
Sep 07 11:40:10.904198 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.904195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a967ae83-c6aa-4939-a1ca-dc93cce4349a-openshift-service-ca\") pod \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") "
Sep 07 11:40:10.904441 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.904225    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn4w9\" (UniqueName: \"kubernetes.io/projected/a967ae83-c6aa-4939-a1ca-dc93cce4349a-kube-api-access-gn4w9\") pod \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") "
Sep 07 11:40:10.904441 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.904256    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-metrics-certs\") pod \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\" (UID: \"a967ae83-c6aa-4939-a1ca-dc93cce4349a\") "
Sep 07 11:40:10.904601 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.904571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a967ae83-c6aa-4939-a1ca-dc93cce4349a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a967ae83-c6aa-4939-a1ca-dc93cce4349a" (UID: "a967ae83-c6aa-4939-a1ca-dc93cce4349a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:40:10.906382 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.906360    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a967ae83-c6aa-4939-a1ca-dc93cce4349a" (UID: "a967ae83-c6aa-4939-a1ca-dc93cce4349a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:40:10.906472 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.906436    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a967ae83-c6aa-4939-a1ca-dc93cce4349a-kube-api-access-gn4w9" (OuterVolumeSpecName: "kube-api-access-gn4w9") pod "a967ae83-c6aa-4939-a1ca-dc93cce4349a" (UID: "a967ae83-c6aa-4939-a1ca-dc93cce4349a"). InnerVolumeSpecName "kube-api-access-gn4w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:40:10.906527 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:10.906464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a967ae83-c6aa-4939-a1ca-dc93cce4349a" (UID: "a967ae83-c6aa-4939-a1ca-dc93cce4349a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:40:11.005880 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.005842    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn4w9\" (UniqueName: \"kubernetes.io/projected/a967ae83-c6aa-4939-a1ca-dc93cce4349a-kube-api-access-gn4w9\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:40:11.005880 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.005878    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:40:11.005880 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.005887    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a967ae83-c6aa-4939-a1ca-dc93cce4349a-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:40:11.006101 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.005894    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a967ae83-c6aa-4939-a1ca-dc93cce4349a-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:40:11.577162 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.577125    2581 generic.go:356] "Generic (PLEG): container finished" podID="a967ae83-c6aa-4939-a1ca-dc93cce4349a" containerID="78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237" exitCode=0
Sep 07 11:40:11.577582 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.577168    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"
Sep 07 11:40:11.577582 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.577212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h" event={"ID":"a967ae83-c6aa-4939-a1ca-dc93cce4349a","Type":"ContainerDied","Data":"78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237"}
Sep 07 11:40:11.577582 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.577260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h" event={"ID":"a967ae83-c6aa-4939-a1ca-dc93cce4349a","Type":"ContainerDied","Data":"ab2dc168a0df802ff744b157a0d948eaeda83ac2acf90077fdde67ff2f5f99a1"}
Sep 07 11:40:11.577582 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.577289    2581 scope.go:117] "RemoveContainer" containerID="78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237"
Sep 07 11:40:11.586168 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.586150    2581 scope.go:117] "RemoveContainer" containerID="78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237"
Sep 07 11:40:11.586458 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:40:11.586438    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237\": container with ID starting with 78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237 not found: ID does not exist" containerID="78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237"
Sep 07 11:40:11.586505 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.586466    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237"} err="failed to get container status \"78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237\": rpc error: code = NotFound desc = could not find container \"78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237\": container with ID starting with 78f221f846b691b50ea655f4bcacbcc89f8dc843347a74a4f6686b4680d2f237 not found: ID does not exist"
Sep 07 11:40:11.593080 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.593060    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"]
Sep 07 11:40:11.601066 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:11.601046    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bf198c-86c788f98c-c6n9h"]
Sep 07 11:40:12.941500 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:12.941467    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a967ae83-c6aa-4939-a1ca-dc93cce4349a" path="/var/lib/kubelet/pods/a967ae83-c6aa-4939-a1ca-dc93cce4349a/volumes"
Sep 07 11:40:35.844161 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.844125    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"]
Sep 07 11:40:35.845309 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.845280    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a967ae83-c6aa-4939-a1ca-dc93cce4349a" containerName="maas-api"
Sep 07 11:40:35.845457 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.845444    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a967ae83-c6aa-4939-a1ca-dc93cce4349a" containerName="maas-api"
Sep 07 11:40:35.845658 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.845646    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a967ae83-c6aa-4939-a1ca-dc93cce4349a" containerName="maas-api"
Sep 07 11:40:35.850371 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.850347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:35.852953 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.852930    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d6b326a7-serving-cert\""
Sep 07 11:40:35.865328 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.865294    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"]
Sep 07 11:40:35.924201 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.924163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-openshift-service-ca\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:35.924201 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.924206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vgk\" (UniqueName: \"kubernetes.io/projected/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-kube-api-access-t8vgk\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:35.924404 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.924279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-maas-api-tls\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:35.924404 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:35.924330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-metrics-certs\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.024699 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.024660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-metrics-certs\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.024892 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.024749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-openshift-service-ca\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.024892 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.024772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vgk\" (UniqueName: \"kubernetes.io/projected/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-kube-api-access-t8vgk\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.024892 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.024847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-maas-api-tls\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.025516 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.025491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-openshift-service-ca\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.027340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.027317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-maas-api-tls\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.028458 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.028432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-metrics-certs\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.038880 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.038498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vgk\" (UniqueName: \"kubernetes.io/projected/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-kube-api-access-t8vgk\") pod \"maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.165112 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.165034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:36.436998 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.436966    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"]
Sep 07 11:40:36.439305 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:40:36.439258    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e4f0d4_1dc5_4e89_a9be_ec2f034ab2be.slice/crio-377ee7a0425c1083fdc111fa7f224885836bbbf0831c803b9b1db0768ef9c258 WatchSource:0}: Error finding container 377ee7a0425c1083fdc111fa7f224885836bbbf0831c803b9b1db0768ef9c258: Status 404 returned error can't find the container with id 377ee7a0425c1083fdc111fa7f224885836bbbf0831c803b9b1db0768ef9c258
Sep 07 11:40:36.700209 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:36.700127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h" event={"ID":"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be","Type":"ContainerStarted","Data":"377ee7a0425c1083fdc111fa7f224885836bbbf0831c803b9b1db0768ef9c258"}
Sep 07 11:40:37.706492 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:37.706449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h" event={"ID":"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be","Type":"ContainerStarted","Data":"5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d"}
Sep 07 11:40:37.706951 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:37.706588    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:37.726052 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:37.726010    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h" podStartSLOduration=2.293780802 podStartE2EDuration="2.725996473s" totalImagesPullingTime="432.215671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:40:37.725228043 +0000 UTC m=+1087.431008735" watchObservedRunningTime="2026-09-07 11:40:37.725996473 +0000 UTC m=+1087.431777162"
Sep 07 11:40:43.717673 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:43.717641    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:40:52.372829 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.372777    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"]
Sep 07 11:40:52.373669 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.373586    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd" podUID="72e4af22-f976-4b4f-b162-298fc6f6b440" containerName="maas-api" containerID="cri-o://6f67138216df9142349bbb82392ee12a586c18def25768f8edad91ffe898f1e4" gracePeriod=30
Sep 07 11:40:52.785686 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.785653    2581 generic.go:356] "Generic (PLEG): container finished" podID="72e4af22-f976-4b4f-b162-298fc6f6b440" containerID="6f67138216df9142349bbb82392ee12a586c18def25768f8edad91ffe898f1e4" exitCode=0
Sep 07 11:40:52.785877 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.785739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd" event={"ID":"72e4af22-f976-4b4f-b162-298fc6f6b440","Type":"ContainerDied","Data":"6f67138216df9142349bbb82392ee12a586c18def25768f8edad91ffe898f1e4"}
Sep 07 11:40:52.838324 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.838299    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:40:52.982516 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.982489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls\") pod \"72e4af22-f976-4b4f-b162-298fc6f6b440\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") "
Sep 07 11:40:52.982695 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.982540    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-metrics-certs\") pod \"72e4af22-f976-4b4f-b162-298fc6f6b440\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") "
Sep 07 11:40:52.982695 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.982576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e4af22-f976-4b4f-b162-298fc6f6b440-openshift-service-ca\") pod \"72e4af22-f976-4b4f-b162-298fc6f6b440\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") "
Sep 07 11:40:52.982695 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.982643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxg55\" (UniqueName: \"kubernetes.io/projected/72e4af22-f976-4b4f-b162-298fc6f6b440-kube-api-access-xxg55\") pod \"72e4af22-f976-4b4f-b162-298fc6f6b440\" (UID: \"72e4af22-f976-4b4f-b162-298fc6f6b440\") "
Sep 07 11:40:52.983041 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.983006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e4af22-f976-4b4f-b162-298fc6f6b440-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72e4af22-f976-4b4f-b162-298fc6f6b440" (UID: "72e4af22-f976-4b4f-b162-298fc6f6b440"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:40:52.984641 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.984608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72e4af22-f976-4b4f-b162-298fc6f6b440" (UID: "72e4af22-f976-4b4f-b162-298fc6f6b440"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:40:52.984641 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.984626    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "72e4af22-f976-4b4f-b162-298fc6f6b440" (UID: "72e4af22-f976-4b4f-b162-298fc6f6b440"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:40:52.984765 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:52.984639    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e4af22-f976-4b4f-b162-298fc6f6b440-kube-api-access-xxg55" (OuterVolumeSpecName: "kube-api-access-xxg55") pod "72e4af22-f976-4b4f-b162-298fc6f6b440" (UID: "72e4af22-f976-4b4f-b162-298fc6f6b440"). InnerVolumeSpecName "kube-api-access-xxg55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:40:53.084259 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.084226    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxg55\" (UniqueName: \"kubernetes.io/projected/72e4af22-f976-4b4f-b162-298fc6f6b440-kube-api-access-xxg55\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:40:53.084259 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.084255    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:40:53.084259 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.084265    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e4af22-f976-4b4f-b162-298fc6f6b440-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:40:53.084456 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.084273    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e4af22-f976-4b4f-b162-298fc6f6b440-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:40:53.792062 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.792033    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"
Sep 07 11:40:53.792062 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.792038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd" event={"ID":"72e4af22-f976-4b4f-b162-298fc6f6b440","Type":"ContainerDied","Data":"55d3ca8287e351ddde1fab060e326351ad49d260351254ba7970de04e5b38e31"}
Sep 07 11:40:53.792488 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.792100    2581 scope.go:117] "RemoveContainer" containerID="6f67138216df9142349bbb82392ee12a586c18def25768f8edad91ffe898f1e4"
Sep 07 11:40:53.814198 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.814172    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"]
Sep 07 11:40:53.820071 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:53.820048    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b7e4c1-78f59b446c-4cjxd"]
Sep 07 11:40:54.941683 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:40:54.941648    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e4af22-f976-4b4f-b162-298fc6f6b440" path="/var/lib/kubelet/pods/72e4af22-f976-4b4f-b162-298fc6f6b440/volumes"
Sep 07 11:41:10.662445 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.662406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"]
Sep 07 11:41:10.663037 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.662708    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl" podUID="56aebcbe-0d94-4fb8-81f6-ca18269732d7" containerName="maas-api" containerID="cri-o://b6eecbc4e48d7bfd38d3bf819a8ab658cf0e536dfd31dcc4d32b5beda24aaf07" gracePeriod=30
Sep 07 11:41:10.885910 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.885876    2581 generic.go:356] "Generic (PLEG): container finished" podID="56aebcbe-0d94-4fb8-81f6-ca18269732d7" containerID="b6eecbc4e48d7bfd38d3bf819a8ab658cf0e536dfd31dcc4d32b5beda24aaf07" exitCode=0
Sep 07 11:41:10.886070 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.885953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl" event={"ID":"56aebcbe-0d94-4fb8-81f6-ca18269732d7","Type":"ContainerDied","Data":"b6eecbc4e48d7bfd38d3bf819a8ab658cf0e536dfd31dcc4d32b5beda24aaf07"}
Sep 07 11:41:10.911560 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.911530    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:41:10.944453 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.944374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg7qd\" (UniqueName: \"kubernetes.io/projected/56aebcbe-0d94-4fb8-81f6-ca18269732d7-kube-api-access-tg7qd\") pod \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") "
Sep 07 11:41:10.944453 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.944424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-metrics-certs\") pod \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") "
Sep 07 11:41:10.946780 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.946745    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56aebcbe-0d94-4fb8-81f6-ca18269732d7" (UID: "56aebcbe-0d94-4fb8-81f6-ca18269732d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:41:10.946931 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:10.946786    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aebcbe-0d94-4fb8-81f6-ca18269732d7-kube-api-access-tg7qd" (OuterVolumeSpecName: "kube-api-access-tg7qd") pod "56aebcbe-0d94-4fb8-81f6-ca18269732d7" (UID: "56aebcbe-0d94-4fb8-81f6-ca18269732d7"). InnerVolumeSpecName "kube-api-access-tg7qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:41:11.045470 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.045412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aebcbe-0d94-4fb8-81f6-ca18269732d7-openshift-service-ca\") pod \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") "
Sep 07 11:41:11.045655 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.045482    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-maas-api-tls\") pod \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\" (UID: \"56aebcbe-0d94-4fb8-81f6-ca18269732d7\") "
Sep 07 11:41:11.045758 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.045745    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7qd\" (UniqueName: \"kubernetes.io/projected/56aebcbe-0d94-4fb8-81f6-ca18269732d7-kube-api-access-tg7qd\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:41:11.045835 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.045766    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:41:11.045897 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.045867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56aebcbe-0d94-4fb8-81f6-ca18269732d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56aebcbe-0d94-4fb8-81f6-ca18269732d7" (UID: "56aebcbe-0d94-4fb8-81f6-ca18269732d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:41:11.047574 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.047551    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56aebcbe-0d94-4fb8-81f6-ca18269732d7" (UID: "56aebcbe-0d94-4fb8-81f6-ca18269732d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:41:11.147098 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.147044    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aebcbe-0d94-4fb8-81f6-ca18269732d7-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:41:11.147098 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.147097    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aebcbe-0d94-4fb8-81f6-ca18269732d7-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:41:11.892703 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.892667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl" event={"ID":"56aebcbe-0d94-4fb8-81f6-ca18269732d7","Type":"ContainerDied","Data":"f4aa2d80ed1d1c94beb57fe7241909318853e14cca7971ced55e9c41ae9eeeb8"}
Sep 07 11:41:11.892703 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.892704    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"
Sep 07 11:41:11.893152 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.892732    2581 scope.go:117] "RemoveContainer" containerID="b6eecbc4e48d7bfd38d3bf819a8ab658cf0e536dfd31dcc4d32b5beda24aaf07"
Sep 07 11:41:11.913338 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.913311    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"]
Sep 07 11:41:11.916998 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:11.916977    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef61c5-5f7ccc95cd-7mjsl"]
Sep 07 11:41:12.942091 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:12.942057    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56aebcbe-0d94-4fb8-81f6-ca18269732d7" path="/var/lib/kubelet/pods/56aebcbe-0d94-4fb8-81f6-ca18269732d7/volumes"
Sep 07 11:41:17.925534 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:17.925500    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"]
Sep 07 11:41:17.926103 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:17.925750    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h" podUID="20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" containerName="maas-api" containerID="cri-o://5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d" gracePeriod=30
Sep 07 11:41:18.176513 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.176462    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:41:18.210923 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.210891    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-maas-api-tls\") pod \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") "
Sep 07 11:41:18.211064 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.211014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-openshift-service-ca\") pod \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") "
Sep 07 11:41:18.211064 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.211050    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-metrics-certs\") pod \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") "
Sep 07 11:41:18.211171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.211082    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8vgk\" (UniqueName: \"kubernetes.io/projected/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-kube-api-access-t8vgk\") pod \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\" (UID: \"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be\") "
Sep 07 11:41:18.211428 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.211396    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" (UID: "20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:41:18.212996 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.212968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" (UID: "20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:41:18.213103 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.213087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-kube-api-access-t8vgk" (OuterVolumeSpecName: "kube-api-access-t8vgk") pod "20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" (UID: "20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be"). InnerVolumeSpecName "kube-api-access-t8vgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:41:18.213170 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.213138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" (UID: "20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:41:18.312222 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.312185    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:41:18.312222 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.312224    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:41:18.312402 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.312240    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vgk\" (UniqueName: \"kubernetes.io/projected/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-kube-api-access-t8vgk\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:41:18.312402 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.312253    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:41:18.930171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.930134    2581 generic.go:356] "Generic (PLEG): container finished" podID="20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" containerID="5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d" exitCode=0
Sep 07 11:41:18.930596 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.930176    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"
Sep 07 11:41:18.930596 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.930212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h" event={"ID":"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be","Type":"ContainerDied","Data":"5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d"}
Sep 07 11:41:18.930596 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.930268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h" event={"ID":"20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be","Type":"ContainerDied","Data":"377ee7a0425c1083fdc111fa7f224885836bbbf0831c803b9b1db0768ef9c258"}
Sep 07 11:41:18.930596 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.930296    2581 scope.go:117] "RemoveContainer" containerID="5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d"
Sep 07 11:41:18.940924 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.940909    2581 scope.go:117] "RemoveContainer" containerID="5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d"
Sep 07 11:41:18.941243 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:41:18.941215    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d\": container with ID starting with 5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d not found: ID does not exist" containerID="5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d"
Sep 07 11:41:18.941322 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.941249    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d"} err="failed to get container status \"5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d\": rpc error: code = NotFound desc = could not find container \"5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d\": container with ID starting with 5a26888d000d234d429ffa9f38daa8624c89adcb49ceb34a74b854c383cb0f8d not found: ID does not exist"
Sep 07 11:41:18.955360 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.955332    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"]
Sep 07 11:41:18.959581 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:18.959552    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d6b326a7-7756f98f4d-r9r7h"]
Sep 07 11:41:20.942181 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:20.942140    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" path="/var/lib/kubelet/pods/20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be/volumes"
Sep 07 11:41:22.966055 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966019    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"]
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966375    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e4af22-f976-4b4f-b162-298fc6f6b440" containerName="maas-api"
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966387    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e4af22-f976-4b4f-b162-298fc6f6b440" containerName="maas-api"
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966395    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" containerName="maas-api"
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966401    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" containerName="maas-api"
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966413    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56aebcbe-0d94-4fb8-81f6-ca18269732d7" containerName="maas-api"
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966421    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56aebcbe-0d94-4fb8-81f6-ca18269732d7" containerName="maas-api"
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966485    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56aebcbe-0d94-4fb8-81f6-ca18269732d7" containerName="maas-api"
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966496    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e4f0d4-1dc5-4e89-a9be-ec2f034ab2be" containerName="maas-api"
Sep 07 11:41:22.966509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.966508    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e4af22-f976-4b4f-b162-298fc6f6b440" containerName="maas-api"
Sep 07 11:41:22.971192 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.971170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:22.973706 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.973681    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6b944b-serving-cert\""
Sep 07 11:41:22.980633 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:22.980608    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"]
Sep 07 11:41:23.052664 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.052624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbk9f\" (UniqueName: \"kubernetes.io/projected/b83e096d-6811-4235-8ee7-af0a54e02316-kube-api-access-pbk9f\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.052983 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.052944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.053187 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.053161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83e096d-6811-4235-8ee7-af0a54e02316-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.053298 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.053260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.154977 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.154938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83e096d-6811-4235-8ee7-af0a54e02316-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.155158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.155003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.155158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.155041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbk9f\" (UniqueName: \"kubernetes.io/projected/b83e096d-6811-4235-8ee7-af0a54e02316-kube-api-access-pbk9f\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.155158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.155090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.155736 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.155704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83e096d-6811-4235-8ee7-af0a54e02316-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.157639 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.157616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.157739 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.157659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.162983 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.162960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbk9f\" (UniqueName: \"kubernetes.io/projected/b83e096d-6811-4235-8ee7-af0a54e02316-kube-api-access-pbk9f\") pod \"maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.282927 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.282840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:23.564480 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.564210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"]
Sep 07 11:41:23.567568 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:41:23.567540    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83e096d_6811_4235_8ee7_af0a54e02316.slice/crio-641bbb019cf813e3dec2b26765a51479bab2c9f8b0de0742c551f39396a76cf5 WatchSource:0}: Error finding container 641bbb019cf813e3dec2b26765a51479bab2c9f8b0de0742c551f39396a76cf5: Status 404 returned error can't find the container with id 641bbb019cf813e3dec2b26765a51479bab2c9f8b0de0742c551f39396a76cf5
Sep 07 11:41:23.962826 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:23.962771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8" event={"ID":"b83e096d-6811-4235-8ee7-af0a54e02316","Type":"ContainerStarted","Data":"641bbb019cf813e3dec2b26765a51479bab2c9f8b0de0742c551f39396a76cf5"}
Sep 07 11:41:24.968842 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:24.968803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8" event={"ID":"b83e096d-6811-4235-8ee7-af0a54e02316","Type":"ContainerStarted","Data":"d58684a4468a1720c294d9807be3a69f030aff26e127c72230209f0921bf2679"}
Sep 07 11:41:24.969243 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:24.968907    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:24.988073 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:24.988025    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8" podStartSLOduration=2.579958132 podStartE2EDuration="2.988011556s" totalImagesPullingTime="408.053424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:24.986931492 +0000 UTC m=+1134.692712180" watchObservedRunningTime="2026-09-07 11:41:24.988011556 +0000 UTC m=+1134.693792245"
Sep 07 11:41:30.979052 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:30.979027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:41:45.601209 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.601166    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"]
Sep 07 11:41:45.606290 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.606269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.608675 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.608654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-0c816d-serving-cert\""
Sep 07 11:41:45.614223 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.614183    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"]
Sep 07 11:41:45.756989 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.756951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.757165 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.757012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6e6ff-fe80-4672-9bc6-c07889622da0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.757165 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.757032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d985g\" (UniqueName: \"kubernetes.io/projected/1fb6e6ff-fe80-4672-9bc6-c07889622da0-kube-api-access-d985g\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.757165 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.757109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.858186 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.858102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.858348 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.858215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.858348 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.858275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6e6ff-fe80-4672-9bc6-c07889622da0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.858348 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.858300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d985g\" (UniqueName: \"kubernetes.io/projected/1fb6e6ff-fe80-4672-9bc6-c07889622da0-kube-api-access-d985g\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.858512 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:41:45.858382    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-serving-cert: secret "maas-api-e2e-shared-a-w2-0c816d-serving-cert" not found
Sep 07 11:41:45.858512 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:41:45.858465    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls podName:1fb6e6ff-fe80-4672-9bc6-c07889622da0 nodeName:}" failed. No retries permitted until 2026-09-07 11:41:46.358442723 +0000 UTC m=+1156.064223404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls") pod "maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc" (UID: "1fb6e6ff-fe80-4672-9bc6-c07889622da0") : secret "maas-api-e2e-shared-a-w2-0c816d-serving-cert" not found
Sep 07 11:41:45.858945 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.858922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6e6ff-fe80-4672-9bc6-c07889622da0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.860827 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.860787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:45.866404 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:45.866384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d985g\" (UniqueName: \"kubernetes.io/projected/1fb6e6ff-fe80-4672-9bc6-c07889622da0-kube-api-access-d985g\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:46.363673 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:46.363639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:46.366047 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:46.366020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:46.519165 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:46.519131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:46.791238 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:46.791053    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"]
Sep 07 11:41:46.793465 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:41:46.793435    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb6e6ff_fe80_4672_9bc6_c07889622da0.slice/crio-3e2f2d3904c329e249bdd4eaa6c34e478a9e125955f55ba93e93b453324d0885 WatchSource:0}: Error finding container 3e2f2d3904c329e249bdd4eaa6c34e478a9e125955f55ba93e93b453324d0885: Status 404 returned error can't find the container with id 3e2f2d3904c329e249bdd4eaa6c34e478a9e125955f55ba93e93b453324d0885
Sep 07 11:41:47.080754 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:47.080679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc" event={"ID":"1fb6e6ff-fe80-4672-9bc6-c07889622da0","Type":"ContainerStarted","Data":"3e2f2d3904c329e249bdd4eaa6c34e478a9e125955f55ba93e93b453324d0885"}
Sep 07 11:41:48.087015 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:48.086983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc" event={"ID":"1fb6e6ff-fe80-4672-9bc6-c07889622da0","Type":"ContainerStarted","Data":"6ff13c6fd72307b254db131268480fc1cbe344e00443b4dea43a86c4bcfad1dc"}
Sep 07 11:41:48.087370 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:48.087096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:48.106544 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:48.106505    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc" podStartSLOduration=2.779703391 podStartE2EDuration="3.106492593s" totalImagesPullingTime="326.789202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:48.105530178 +0000 UTC m=+1157.811310891" watchObservedRunningTime="2026-09-07 11:41:48.106492593 +0000 UTC m=+1157.812273282"
Sep 07 11:41:51.295234 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.295187    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"]
Sep 07 11:41:51.299675 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.299653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.302026 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.302005    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0f0a2f-serving-cert\""
Sep 07 11:41:51.307432 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.307406    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"]
Sep 07 11:41:51.411853 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.411818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.412020 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.411858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeba831-43ba-4e67-bf30-07adb4a3b224-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.412020 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.411994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq4k9\" (UniqueName: \"kubernetes.io/projected/edeba831-43ba-4e67-bf30-07adb4a3b224-kube-api-access-bq4k9\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.412103 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.412079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.513125 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.513093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.513274 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.513147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.513323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.513263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeba831-43ba-4e67-bf30-07adb4a3b224-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.513370 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.513355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq4k9\" (UniqueName: \"kubernetes.io/projected/edeba831-43ba-4e67-bf30-07adb4a3b224-kube-api-access-bq4k9\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.513902 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.513879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeba831-43ba-4e67-bf30-07adb4a3b224-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.515502 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.515484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.515572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.515522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.523357 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.523338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq4k9\" (UniqueName: \"kubernetes.io/projected/edeba831-43ba-4e67-bf30-07adb4a3b224-kube-api-access-bq4k9\") pod \"maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.612482 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.612419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:51.883427 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:51.883224    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"]
Sep 07 11:41:51.885856 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:41:51.885830    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedeba831_43ba_4e67_bf30_07adb4a3b224.slice/crio-3a514146c5a723a75b61bf51624a831008e16bdf9d769ecacdb010ac6b368e70 WatchSource:0}: Error finding container 3a514146c5a723a75b61bf51624a831008e16bdf9d769ecacdb010ac6b368e70: Status 404 returned error can't find the container with id 3a514146c5a723a75b61bf51624a831008e16bdf9d769ecacdb010ac6b368e70
Sep 07 11:41:52.109105 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:52.109072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5" event={"ID":"edeba831-43ba-4e67-bf30-07adb4a3b224","Type":"ContainerStarted","Data":"3a514146c5a723a75b61bf51624a831008e16bdf9d769ecacdb010ac6b368e70"}
Sep 07 11:41:53.116192 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:53.116101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5" event={"ID":"edeba831-43ba-4e67-bf30-07adb4a3b224","Type":"ContainerStarted","Data":"9debe618dab8fb986214bd64111b6063136b7ed7d4123428517d35d1228f80d5"}
Sep 07 11:41:53.116526 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:53.116184    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:41:53.148978 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:53.148930    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5" podStartSLOduration=1.796629168 podStartE2EDuration="2.148914952s" totalImagesPullingTime="352.285784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:53.147483367 +0000 UTC m=+1162.853264060" watchObservedRunningTime="2026-09-07 11:41:53.148914952 +0000 UTC m=+1162.854695704"
Sep 07 11:41:54.098918 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:54.098888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:41:59.125559 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:41:59.125521    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:42:14.504998 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.504962    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"]
Sep 07 11:42:14.509092 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.509069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.511251 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.511227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2195e0-serving-cert\""
Sep 07 11:42:14.521371 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.521309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"]
Sep 07 11:42:14.628505 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.628469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.628832 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.628530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7vs\" (UniqueName: \"kubernetes.io/projected/a39fb8e4-315a-4341-86a7-c7bafc2fe461-kube-api-access-lm7vs\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.628832 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.628589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.628832 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.628621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39fb8e4-315a-4341-86a7-c7bafc2fe461-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.729957 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.729921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.730126 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.729976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7vs\" (UniqueName: \"kubernetes.io/projected/a39fb8e4-315a-4341-86a7-c7bafc2fe461-kube-api-access-lm7vs\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.730126 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.730012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.730126 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.730039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39fb8e4-315a-4341-86a7-c7bafc2fe461-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.730721 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.730699    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39fb8e4-315a-4341-86a7-c7bafc2fe461-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.732532 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.732504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.732621 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.732514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.738643 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.738621    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7vs\" (UniqueName: \"kubernetes.io/projected/a39fb8e4-315a-4341-86a7-c7bafc2fe461-kube-api-access-lm7vs\") pod \"maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:14.820860 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:14.820764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:15.097302 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:15.097270    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"]
Sep 07 11:42:15.098958 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:42:15.098927    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39fb8e4_315a_4341_86a7_c7bafc2fe461.slice/crio-b8e2a8c6f42fe4b658f8590e3c669605bda267eeefb639206e9a2bb00719aec6 WatchSource:0}: Error finding container b8e2a8c6f42fe4b658f8590e3c669605bda267eeefb639206e9a2bb00719aec6: Status 404 returned error can't find the container with id b8e2a8c6f42fe4b658f8590e3c669605bda267eeefb639206e9a2bb00719aec6
Sep 07 11:42:15.227028 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:15.226988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb" event={"ID":"a39fb8e4-315a-4341-86a7-c7bafc2fe461","Type":"ContainerStarted","Data":"b8e2a8c6f42fe4b658f8590e3c669605bda267eeefb639206e9a2bb00719aec6"}
Sep 07 11:42:16.233155 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:16.233116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb" event={"ID":"a39fb8e4-315a-4341-86a7-c7bafc2fe461","Type":"ContainerStarted","Data":"052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802"}
Sep 07 11:42:16.233634 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:16.233231    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:16.251531 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:16.251482    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb" podStartSLOduration=1.934018437 podStartE2EDuration="2.251468369s" totalImagesPullingTime="317.449932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:42:16.250772775 +0000 UTC m=+1185.956553466" watchObservedRunningTime="2026-09-07 11:42:16.251468369 +0000 UTC m=+1185.957249085"
Sep 07 11:42:22.245166 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:22.245135    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:42:55.553432 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.553385    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"]
Sep 07 11:42:55.557921 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.557900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.560978 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.560945    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8108f8fd-serving-cert\""
Sep 07 11:42:55.568405 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.568381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"]
Sep 07 11:42:55.597051 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.597020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9641f-b982-423a-ada5-5309004b4c1d-openshift-service-ca\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.597223 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.597054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/41d9641f-b982-423a-ada5-5309004b4c1d-kube-api-access-dvkz4\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.597223 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.597077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.597321 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.597250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-metrics-certs\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.698603 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.698573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9641f-b982-423a-ada5-5309004b4c1d-openshift-service-ca\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.698786 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.698616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/41d9641f-b982-423a-ada5-5309004b4c1d-kube-api-access-dvkz4\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.698786 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.698641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.698786 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.698711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-metrics-certs\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.698998 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:42:55.698845    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-serving-cert: secret "maas-api-e2e-mt-8108f8fd-serving-cert" not found
Sep 07 11:42:55.698998 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:42:55.698930    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls podName:41d9641f-b982-423a-ada5-5309004b4c1d nodeName:}" failed. No retries permitted until 2026-09-07 11:42:56.198909232 +0000 UTC m=+1225.904689904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls") pod "maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p" (UID: "41d9641f-b982-423a-ada5-5309004b4c1d") : secret "maas-api-e2e-mt-8108f8fd-serving-cert" not found
Sep 07 11:42:55.699407 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.699385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9641f-b982-423a-ada5-5309004b4c1d-openshift-service-ca\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.701206 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.701188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-metrics-certs\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:55.711490 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:55.711469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/41d9641f-b982-423a-ada5-5309004b4c1d-kube-api-access-dvkz4\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:56.202313 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:56.202278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:56.204597 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:56.204571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls\") pod \"maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:56.468997 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:56.468965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:56.748323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:56.748290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"]
Sep 07 11:42:56.751013 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:42:56.750979    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d9641f_b982_423a_ada5_5309004b4c1d.slice/crio-db40c759adef235b587fd0dc45929d39fe7ef13885b0cdd6e844a5f8217628ab WatchSource:0}: Error finding container db40c759adef235b587fd0dc45929d39fe7ef13885b0cdd6e844a5f8217628ab: Status 404 returned error can't find the container with id db40c759adef235b587fd0dc45929d39fe7ef13885b0cdd6e844a5f8217628ab
Sep 07 11:42:57.451039 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:57.450959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p" event={"ID":"41d9641f-b982-423a-ada5-5309004b4c1d","Type":"ContainerStarted","Data":"44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690"}
Sep 07 11:42:57.451039 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:57.451020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p" event={"ID":"41d9641f-b982-423a-ada5-5309004b4c1d","Type":"ContainerStarted","Data":"db40c759adef235b587fd0dc45929d39fe7ef13885b0cdd6e844a5f8217628ab"}
Sep 07 11:42:57.451234 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:57.451072    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:42:57.476087 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:42:57.476032    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p" podStartSLOduration=2.03387805 podStartE2EDuration="2.476012329s" totalImagesPullingTime="442.134279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:42:57.473516374 +0000 UTC m=+1227.179297064" watchObservedRunningTime="2026-09-07 11:42:57.476012329 +0000 UTC m=+1227.181793020"
Sep 07 11:43:03.461459 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:43:03.461423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:44:40.482076 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.482037    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"]
Sep 07 11:44:40.482658 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.482284    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p" podUID="41d9641f-b982-423a-ada5-5309004b4c1d" containerName="maas-api" containerID="cri-o://44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690" gracePeriod=30
Sep 07 11:44:40.751643 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.751618    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:44:40.864431 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.864397    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9641f-b982-423a-ada5-5309004b4c1d-openshift-service-ca\") pod \"41d9641f-b982-423a-ada5-5309004b4c1d\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") "
Sep 07 11:44:40.864574 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.864502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/41d9641f-b982-423a-ada5-5309004b4c1d-kube-api-access-dvkz4\") pod \"41d9641f-b982-423a-ada5-5309004b4c1d\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") "
Sep 07 11:44:40.864574 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.864526    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-metrics-certs\") pod \"41d9641f-b982-423a-ada5-5309004b4c1d\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") "
Sep 07 11:44:40.864662 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.864580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls\") pod \"41d9641f-b982-423a-ada5-5309004b4c1d\" (UID: \"41d9641f-b982-423a-ada5-5309004b4c1d\") "
Sep 07 11:44:40.864802 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.864760    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d9641f-b982-423a-ada5-5309004b4c1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41d9641f-b982-423a-ada5-5309004b4c1d" (UID: "41d9641f-b982-423a-ada5-5309004b4c1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:44:40.864945 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.864922    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9641f-b982-423a-ada5-5309004b4c1d-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:44:40.866746 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.866719    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41d9641f-b982-423a-ada5-5309004b4c1d" (UID: "41d9641f-b982-423a-ada5-5309004b4c1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:44:40.866746 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.866741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41d9641f-b982-423a-ada5-5309004b4c1d" (UID: "41d9641f-b982-423a-ada5-5309004b4c1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:44:40.866949 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.866851    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d9641f-b982-423a-ada5-5309004b4c1d-kube-api-access-dvkz4" (OuterVolumeSpecName: "kube-api-access-dvkz4") pod "41d9641f-b982-423a-ada5-5309004b4c1d" (UID: "41d9641f-b982-423a-ada5-5309004b4c1d"). InnerVolumeSpecName "kube-api-access-dvkz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:44:40.959972 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.959936    2581 generic.go:356] "Generic (PLEG): container finished" podID="41d9641f-b982-423a-ada5-5309004b4c1d" containerID="44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690" exitCode=0
Sep 07 11:44:40.960093 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.959981    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"
Sep 07 11:44:40.960093 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.960007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p" event={"ID":"41d9641f-b982-423a-ada5-5309004b4c1d","Type":"ContainerDied","Data":"44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690"}
Sep 07 11:44:40.960093 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.960054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p" event={"ID":"41d9641f-b982-423a-ada5-5309004b4c1d","Type":"ContainerDied","Data":"db40c759adef235b587fd0dc45929d39fe7ef13885b0cdd6e844a5f8217628ab"}
Sep 07 11:44:40.960093 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.960077    2581 scope.go:117] "RemoveContainer" containerID="44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690"
Sep 07 11:44:40.965824 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.965784    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:44:40.965921 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.965826    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/41d9641f-b982-423a-ada5-5309004b4c1d-kube-api-access-dvkz4\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:44:40.965921 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.965836    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9641f-b982-423a-ada5-5309004b4c1d-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:44:40.968954 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.968932    2581 scope.go:117] "RemoveContainer" containerID="44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690"
Sep 07 11:44:40.969221 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:44:40.969204    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690\": container with ID starting with 44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690 not found: ID does not exist" containerID="44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690"
Sep 07 11:44:40.969291 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.969234    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690"} err="failed to get container status \"44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690\": rpc error: code = NotFound desc = could not find container \"44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690\": container with ID starting with 44b324c5025d588c0d496f78813423200bd3bd7893b467a9155f802223bdd690 not found: ID does not exist"
Sep 07 11:44:40.977207 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.977182    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"]
Sep 07 11:44:40.980322 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:40.980301    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8108f8fd-7dc86cbd5d-5g62p"]
Sep 07 11:44:42.942776 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:44:42.942731    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d9641f-b982-423a-ada5-5309004b4c1d" path="/var/lib/kubelet/pods/41d9641f-b982-423a-ada5-5309004b4c1d/volumes"
Sep 07 11:46:32.780294 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.780258    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"]
Sep 07 11:46:32.781250 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.781224    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d9641f-b982-423a-ada5-5309004b4c1d" containerName="maas-api"
Sep 07 11:46:32.781250 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.781249    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d9641f-b982-423a-ada5-5309004b4c1d" containerName="maas-api"
Sep 07 11:46:32.781454 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.781319    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d9641f-b982-423a-ada5-5309004b4c1d" containerName="maas-api"
Sep 07 11:46:32.784410 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.784390    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.787242 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.787221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2cfab4-serving-cert\""
Sep 07 11:46:32.796484 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.796462    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"]
Sep 07 11:46:32.865856 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.865784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.866020 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.865870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-metrics-certs\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.866020 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.865895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-openshift-service-ca\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.866020 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.865922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bv7\" (UniqueName: \"kubernetes.io/projected/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-kube-api-access-g9bv7\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.966616 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.966583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-metrics-certs\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.966813 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.966628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-openshift-service-ca\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.966813 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.966667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bv7\" (UniqueName: \"kubernetes.io/projected/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-kube-api-access-g9bv7\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.966813 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.966770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.967214 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:46:32.967185    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-serving-cert: secret "maas-api-e2e-api-a-2cfab4-serving-cert" not found
Sep 07 11:46:32.967320 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:46:32.967295    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls podName:f6e1ee43-8a9d-4a90-91d4-e830b1e86906 nodeName:}" failed. No retries permitted until 2026-09-07 11:46:33.467270641 +0000 UTC m=+1443.173051313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls") pod "maas-api-e2e-api-a-2cfab4-5674658554-cmqlg" (UID: "f6e1ee43-8a9d-4a90-91d4-e830b1e86906") : secret "maas-api-e2e-api-a-2cfab4-serving-cert" not found
Sep 07 11:46:32.967423 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.967402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-openshift-service-ca\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.969250 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.969225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-metrics-certs\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:32.978102 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:32.978083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bv7\" (UniqueName: \"kubernetes.io/projected/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-kube-api-access-g9bv7\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:33.472352 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:33.472317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:33.474902 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:33.474871    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls\") pod \"maas-api-e2e-api-a-2cfab4-5674658554-cmqlg\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:33.696466 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:33.696427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:33.974087 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:33.974060    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"]
Sep 07 11:46:33.976268 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:46:33.976244    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e1ee43_8a9d_4a90_91d4_e830b1e86906.slice/crio-b80fca07e86ba3d04e4353e1a1d11484d77622e3265a07cc758d2be2327eb904 WatchSource:0}: Error finding container b80fca07e86ba3d04e4353e1a1d11484d77622e3265a07cc758d2be2327eb904: Status 404 returned error can't find the container with id b80fca07e86ba3d04e4353e1a1d11484d77622e3265a07cc758d2be2327eb904
Sep 07 11:46:33.977820 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:33.977777    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:46:34.539786 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:34.539748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg" event={"ID":"f6e1ee43-8a9d-4a90-91d4-e830b1e86906","Type":"ContainerStarted","Data":"3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709"}
Sep 07 11:46:34.539979 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:34.539866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg" event={"ID":"f6e1ee43-8a9d-4a90-91d4-e830b1e86906","Type":"ContainerStarted","Data":"b80fca07e86ba3d04e4353e1a1d11484d77622e3265a07cc758d2be2327eb904"}
Sep 07 11:46:34.539979 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:34.539915    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:46:34.559556 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:34.559512    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg" podStartSLOduration=2.117896734 podStartE2EDuration="2.559498677s" totalImagesPullingTime="441.601943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:46:34.557496534 +0000 UTC m=+1444.263277235" watchObservedRunningTime="2026-09-07 11:46:34.559498677 +0000 UTC m=+1444.265279367"
Sep 07 11:46:40.550500 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:46:40.550473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:47:02.404189 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.404154    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"]
Sep 07 11:47:02.407873 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.407853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.410265 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.410234    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f57a16-serving-cert\""
Sep 07 11:47:02.417341 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.417316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"]
Sep 07 11:47:02.549129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.549090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d1580-7203-4fbf-9f21-1cb58b51793e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.549129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.549126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdh49\" (UniqueName: \"kubernetes.io/projected/6e9d1580-7203-4fbf-9f21-1cb58b51793e-kube-api-access-qdh49\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.549332 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.549168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-metrics-certs\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.549332 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.549218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-maas-api-tls\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.650502 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.650469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-metrics-certs\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.650657 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.650533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-maas-api-tls\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.650657 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.650614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d1580-7203-4fbf-9f21-1cb58b51793e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.650657 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.650639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdh49\" (UniqueName: \"kubernetes.io/projected/6e9d1580-7203-4fbf-9f21-1cb58b51793e-kube-api-access-qdh49\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.651324 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.651293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d1580-7203-4fbf-9f21-1cb58b51793e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.653011 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.652990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-maas-api-tls\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.653118 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.653063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-metrics-certs\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.665585 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.665525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdh49\" (UniqueName: \"kubernetes.io/projected/6e9d1580-7203-4fbf-9f21-1cb58b51793e-kube-api-access-qdh49\") pod \"maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.719121 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.719092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:02.998484 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:02.998457    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"]
Sep 07 11:47:03.000467 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:47:03.000438    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9d1580_7203_4fbf_9f21_1cb58b51793e.slice/crio-0db0c15ab2f37940ad3ac74a36a70100b7e636f44696da0ee8a3077f7797432b WatchSource:0}: Error finding container 0db0c15ab2f37940ad3ac74a36a70100b7e636f44696da0ee8a3077f7797432b: Status 404 returned error can't find the container with id 0db0c15ab2f37940ad3ac74a36a70100b7e636f44696da0ee8a3077f7797432b
Sep 07 11:47:03.691463 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:03.691417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq" event={"ID":"6e9d1580-7203-4fbf-9f21-1cb58b51793e","Type":"ContainerStarted","Data":"432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64"}
Sep 07 11:47:03.691935 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:03.691490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq" event={"ID":"6e9d1580-7203-4fbf-9f21-1cb58b51793e","Type":"ContainerStarted","Data":"0db0c15ab2f37940ad3ac74a36a70100b7e636f44696da0ee8a3077f7797432b"}
Sep 07 11:47:03.691935 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:03.691575    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:03.710775 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:03.710712    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq" podStartSLOduration=1.119677394 podStartE2EDuration="1.710695739s" totalImagesPullingTime="591.018345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:47:03.710367038 +0000 UTC m=+1473.416147729" watchObservedRunningTime="2026-09-07 11:47:03.710695739 +0000 UTC m=+1473.416476429"
Sep 07 11:47:09.701527 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:09.701492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:47:45.185948 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.185912    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"]
Sep 07 11:47:45.186543 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.186104    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg" podUID="f6e1ee43-8a9d-4a90-91d4-e830b1e86906" containerName="maas-api" containerID="cri-o://3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709" gracePeriod=30
Sep 07 11:47:45.455749 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.455720    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:47:45.541137 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.541103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls\") pod \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") "
Sep 07 11:47:45.541320 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.541280    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-metrics-certs\") pod \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") "
Sep 07 11:47:45.541389 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.541331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9bv7\" (UniqueName: \"kubernetes.io/projected/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-kube-api-access-g9bv7\") pod \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") "
Sep 07 11:47:45.541450 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.541403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-openshift-service-ca\") pod \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\" (UID: \"f6e1ee43-8a9d-4a90-91d4-e830b1e86906\") "
Sep 07 11:47:45.542124 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.542092    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6e1ee43-8a9d-4a90-91d4-e830b1e86906" (UID: "f6e1ee43-8a9d-4a90-91d4-e830b1e86906"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:47:45.544392 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.544366    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6e1ee43-8a9d-4a90-91d4-e830b1e86906" (UID: "f6e1ee43-8a9d-4a90-91d4-e830b1e86906"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:47:45.544526 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.544501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6e1ee43-8a9d-4a90-91d4-e830b1e86906" (UID: "f6e1ee43-8a9d-4a90-91d4-e830b1e86906"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:47:45.544747 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.544722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-kube-api-access-g9bv7" (OuterVolumeSpecName: "kube-api-access-g9bv7") pod "f6e1ee43-8a9d-4a90-91d4-e830b1e86906" (UID: "f6e1ee43-8a9d-4a90-91d4-e830b1e86906"). InnerVolumeSpecName "kube-api-access-g9bv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:47:45.642424 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.642389    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:47:45.642424 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.642417    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:47:45.642424 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.642426    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bv7\" (UniqueName: \"kubernetes.io/projected/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-kube-api-access-g9bv7\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:47:45.642645 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.642436    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e1ee43-8a9d-4a90-91d4-e830b1e86906-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:47:45.908851 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.908775    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6e1ee43-8a9d-4a90-91d4-e830b1e86906" containerID="3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709" exitCode=0
Sep 07 11:47:45.908851 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.908838    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"
Sep 07 11:47:45.909045 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.908867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg" event={"ID":"f6e1ee43-8a9d-4a90-91d4-e830b1e86906","Type":"ContainerDied","Data":"3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709"}
Sep 07 11:47:45.909045 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.908950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg" event={"ID":"f6e1ee43-8a9d-4a90-91d4-e830b1e86906","Type":"ContainerDied","Data":"b80fca07e86ba3d04e4353e1a1d11484d77622e3265a07cc758d2be2327eb904"}
Sep 07 11:47:45.909045 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.908974    2581 scope.go:117] "RemoveContainer" containerID="3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709"
Sep 07 11:47:45.918554 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.918536    2581 scope.go:117] "RemoveContainer" containerID="3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709"
Sep 07 11:47:45.918877 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:47:45.918858    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709\": container with ID starting with 3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709 not found: ID does not exist" containerID="3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709"
Sep 07 11:47:45.918952 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.918885    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709"} err="failed to get container status \"3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709\": rpc error: code = NotFound desc = could not find container \"3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709\": container with ID starting with 3168a3d6da961ebd51529a4d165c6ccd90fb1f8c11ad9f55cad3a31d3eebc709 not found: ID does not exist"
Sep 07 11:47:45.929889 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.929867    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"]
Sep 07 11:47:45.934119 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:45.934094    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cfab4-5674658554-cmqlg"]
Sep 07 11:47:46.941241 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:47:46.941204    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e1ee43-8a9d-4a90-91d4-e830b1e86906" path="/var/lib/kubelet/pods/f6e1ee43-8a9d-4a90-91d4-e830b1e86906/volumes"
Sep 07 11:48:07.334693 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.334653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"]
Sep 07 11:48:07.335171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.335091    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e1ee43-8a9d-4a90-91d4-e830b1e86906" containerName="maas-api"
Sep 07 11:48:07.335171 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.335102    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e1ee43-8a9d-4a90-91d4-e830b1e86906" containerName="maas-api"
Sep 07 11:48:07.335295 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.335175    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e1ee43-8a9d-4a90-91d4-e830b1e86906" containerName="maas-api"
Sep 07 11:48:07.338555 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.338529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.341198 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.341171    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-984178-serving-cert\""
Sep 07 11:48:07.352462 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.352438    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"]
Sep 07 11:48:07.444263 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.444228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66406c3-ffc7-4b55-a92d-acd69954cbdf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.444467 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.444287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.444467 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.444309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8l6t\" (UniqueName: \"kubernetes.io/projected/b66406c3-ffc7-4b55-a92d-acd69954cbdf-kube-api-access-t8l6t\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.444467 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.444359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-metrics-certs\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.545589 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.545548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-metrics-certs\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.545778 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.545669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66406c3-ffc7-4b55-a92d-acd69954cbdf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.545778 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.545735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.545778 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.545761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8l6t\" (UniqueName: \"kubernetes.io/projected/b66406c3-ffc7-4b55-a92d-acd69954cbdf-kube-api-access-t8l6t\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.545998 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:48:07.545871    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-serving-cert: secret "maas-api-e2e-ipp-a-984178-serving-cert" not found
Sep 07 11:48:07.545998 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:48:07.545978    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls podName:b66406c3-ffc7-4b55-a92d-acd69954cbdf nodeName:}" failed. No retries permitted until 2026-09-07 11:48:08.045954245 +0000 UTC m=+1537.751734930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls") pod "maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz" (UID: "b66406c3-ffc7-4b55-a92d-acd69954cbdf") : secret "maas-api-e2e-ipp-a-984178-serving-cert" not found
Sep 07 11:48:07.546420 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.546401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66406c3-ffc7-4b55-a92d-acd69954cbdf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.548158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.548132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-metrics-certs\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:07.556533 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:07.556514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8l6t\" (UniqueName: \"kubernetes.io/projected/b66406c3-ffc7-4b55-a92d-acd69954cbdf-kube-api-access-t8l6t\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:08.050438 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.050404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:08.052742 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.052712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:08.224900 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.224860    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"]
Sep 07 11:48:08.225163 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.225112    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq" podUID="6e9d1580-7203-4fbf-9f21-1cb58b51793e" containerName="maas-api" containerID="cri-o://432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64" gracePeriod=30
Sep 07 11:48:08.252740 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.252710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:08.489552 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.489525    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:48:08.547124 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.547092    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"]
Sep 07 11:48:08.548674 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:48:08.548645    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66406c3_ffc7_4b55_a92d_acd69954cbdf.slice/crio-9e4cd26dda16624f35ab51c45bf560b9a672ef66a50c3cc6a17b67fbea734927 WatchSource:0}: Error finding container 9e4cd26dda16624f35ab51c45bf560b9a672ef66a50c3cc6a17b67fbea734927: Status 404 returned error can't find the container with id 9e4cd26dda16624f35ab51c45bf560b9a672ef66a50c3cc6a17b67fbea734927
Sep 07 11:48:08.556557 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.556533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-metrics-certs\") pod \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") "
Sep 07 11:48:08.556654 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.556607    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d1580-7203-4fbf-9f21-1cb58b51793e-openshift-service-ca\") pod \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") "
Sep 07 11:48:08.556753 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.556738    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-maas-api-tls\") pod \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") "
Sep 07 11:48:08.557234 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.556770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdh49\" (UniqueName: \"kubernetes.io/projected/6e9d1580-7203-4fbf-9f21-1cb58b51793e-kube-api-access-qdh49\") pod \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\" (UID: \"6e9d1580-7203-4fbf-9f21-1cb58b51793e\") "
Sep 07 11:48:08.557234 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.557125    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9d1580-7203-4fbf-9f21-1cb58b51793e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e9d1580-7203-4fbf-9f21-1cb58b51793e" (UID: "6e9d1580-7203-4fbf-9f21-1cb58b51793e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:08.559738 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.559706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e9d1580-7203-4fbf-9f21-1cb58b51793e" (UID: "6e9d1580-7203-4fbf-9f21-1cb58b51793e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:48:08.560011 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.559986    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e9d1580-7203-4fbf-9f21-1cb58b51793e" (UID: "6e9d1580-7203-4fbf-9f21-1cb58b51793e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:48:08.560068 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.560038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9d1580-7203-4fbf-9f21-1cb58b51793e-kube-api-access-qdh49" (OuterVolumeSpecName: "kube-api-access-qdh49") pod "6e9d1580-7203-4fbf-9f21-1cb58b51793e" (UID: "6e9d1580-7203-4fbf-9f21-1cb58b51793e"). InnerVolumeSpecName "kube-api-access-qdh49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:08.657814 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.657754    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:08.657814 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.657786    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdh49\" (UniqueName: \"kubernetes.io/projected/6e9d1580-7203-4fbf-9f21-1cb58b51793e-kube-api-access-qdh49\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:08.657814 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.657812    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d1580-7203-4fbf-9f21-1cb58b51793e-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:08.658083 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:08.657826    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d1580-7203-4fbf-9f21-1cb58b51793e-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:09.029083 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.029049    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e9d1580-7203-4fbf-9f21-1cb58b51793e" containerID="432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64" exitCode=0
Sep 07 11:48:09.029301 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.029111    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"
Sep 07 11:48:09.029301 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.029151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq" event={"ID":"6e9d1580-7203-4fbf-9f21-1cb58b51793e","Type":"ContainerDied","Data":"432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64"}
Sep 07 11:48:09.029301 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.029218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq" event={"ID":"6e9d1580-7203-4fbf-9f21-1cb58b51793e","Type":"ContainerDied","Data":"0db0c15ab2f37940ad3ac74a36a70100b7e636f44696da0ee8a3077f7797432b"}
Sep 07 11:48:09.029301 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.029248    2581 scope.go:117] "RemoveContainer" containerID="432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64"
Sep 07 11:48:09.031371 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.031329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz" event={"ID":"b66406c3-ffc7-4b55-a92d-acd69954cbdf","Type":"ContainerStarted","Data":"abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e"}
Sep 07 11:48:09.031490 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.031386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz" event={"ID":"b66406c3-ffc7-4b55-a92d-acd69954cbdf","Type":"ContainerStarted","Data":"9e4cd26dda16624f35ab51c45bf560b9a672ef66a50c3cc6a17b67fbea734927"}
Sep 07 11:48:09.031625 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.031603    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:09.039450 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.039420    2581 scope.go:117] "RemoveContainer" containerID="432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64"
Sep 07 11:48:09.039806 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:48:09.039765    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64\": container with ID starting with 432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64 not found: ID does not exist" containerID="432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64"
Sep 07 11:48:09.039918 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.039812    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64"} err="failed to get container status \"432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64\": rpc error: code = NotFound desc = could not find container \"432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64\": container with ID starting with 432b141ebaa56e45625952ac02187f31f58301aa03f509fc6f1393864cfdfa64 not found: ID does not exist"
Sep 07 11:48:09.063643 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.063575    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz" podStartSLOduration=1.729012335 podStartE2EDuration="2.063557097s" totalImagesPullingTime="334.544762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:48:09.049643218 +0000 UTC m=+1538.755423907" watchObservedRunningTime="2026-09-07 11:48:09.063557097 +0000 UTC m=+1538.769337787"
Sep 07 11:48:09.064249 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.064229    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"]
Sep 07 11:48:09.067877 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:09.067847    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f57a16-7b749fc969-nqfnq"]
Sep 07 11:48:10.943392 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:10.943356    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e9d1580-7203-4fbf-9f21-1cb58b51793e" path="/var/lib/kubelet/pods/6e9d1580-7203-4fbf-9f21-1cb58b51793e/volumes"
Sep 07 11:48:15.043596 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:15.043569    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:48:25.900229 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:25.900190    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"]
Sep 07 11:48:25.900822 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:25.900437    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc" podUID="1fb6e6ff-fe80-4672-9bc6-c07889622da0" containerName="maas-api" containerID="cri-o://6ff13c6fd72307b254db131268480fc1cbe344e00443b4dea43a86c4bcfad1dc" gracePeriod=30
Sep 07 11:48:26.123478 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.123443    2581 generic.go:356] "Generic (PLEG): container finished" podID="1fb6e6ff-fe80-4672-9bc6-c07889622da0" containerID="6ff13c6fd72307b254db131268480fc1cbe344e00443b4dea43a86c4bcfad1dc" exitCode=0
Sep 07 11:48:26.123688 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.123500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc" event={"ID":"1fb6e6ff-fe80-4672-9bc6-c07889622da0","Type":"ContainerDied","Data":"6ff13c6fd72307b254db131268480fc1cbe344e00443b4dea43a86c4bcfad1dc"}
Sep 07 11:48:26.160896 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.160834    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:48:26.322571 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.322533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6e6ff-fe80-4672-9bc6-c07889622da0-openshift-service-ca\") pod \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") "
Sep 07 11:48:26.322848 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.322585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d985g\" (UniqueName: \"kubernetes.io/projected/1fb6e6ff-fe80-4672-9bc6-c07889622da0-kube-api-access-d985g\") pod \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") "
Sep 07 11:48:26.322848 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.322621    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-metrics-certs\") pod \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") "
Sep 07 11:48:26.322848 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.322676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls\") pod \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\" (UID: \"1fb6e6ff-fe80-4672-9bc6-c07889622da0\") "
Sep 07 11:48:26.323066 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.323027    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb6e6ff-fe80-4672-9bc6-c07889622da0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1fb6e6ff-fe80-4672-9bc6-c07889622da0" (UID: "1fb6e6ff-fe80-4672-9bc6-c07889622da0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:26.324906 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.324867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1fb6e6ff-fe80-4672-9bc6-c07889622da0" (UID: "1fb6e6ff-fe80-4672-9bc6-c07889622da0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:48:26.325024 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.324904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1fb6e6ff-fe80-4672-9bc6-c07889622da0" (UID: "1fb6e6ff-fe80-4672-9bc6-c07889622da0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:48:26.325068 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.325018    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb6e6ff-fe80-4672-9bc6-c07889622da0-kube-api-access-d985g" (OuterVolumeSpecName: "kube-api-access-d985g") pod "1fb6e6ff-fe80-4672-9bc6-c07889622da0" (UID: "1fb6e6ff-fe80-4672-9bc6-c07889622da0"). InnerVolumeSpecName "kube-api-access-d985g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:26.423574 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.423495    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6e6ff-fe80-4672-9bc6-c07889622da0-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:26.423574 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.423526    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d985g\" (UniqueName: \"kubernetes.io/projected/1fb6e6ff-fe80-4672-9bc6-c07889622da0-kube-api-access-d985g\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:26.423574 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.423537    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:26.423574 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:26.423546    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb6e6ff-fe80-4672-9bc6-c07889622da0-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:27.129940 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:27.129908    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"
Sep 07 11:48:27.130386 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:27.129908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc" event={"ID":"1fb6e6ff-fe80-4672-9bc6-c07889622da0","Type":"ContainerDied","Data":"3e2f2d3904c329e249bdd4eaa6c34e478a9e125955f55ba93e93b453324d0885"}
Sep 07 11:48:27.130386 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:27.130054    2581 scope.go:117] "RemoveContainer" containerID="6ff13c6fd72307b254db131268480fc1cbe344e00443b4dea43a86c4bcfad1dc"
Sep 07 11:48:27.151037 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:27.151004    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"]
Sep 07 11:48:27.154778 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:27.154750    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0c816d-779dcb974c-8jjgc"]
Sep 07 11:48:28.941919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:28.941886    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb6e6ff-fe80-4672-9bc6-c07889622da0" path="/var/lib/kubelet/pods/1fb6e6ff-fe80-4672-9bc6-c07889622da0/volumes"
Sep 07 11:48:36.996345 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:36.996309    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"]
Sep 07 11:48:36.997297 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:36.997274    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb6e6ff-fe80-4672-9bc6-c07889622da0" containerName="maas-api"
Sep 07 11:48:36.997297 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:36.997298    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb6e6ff-fe80-4672-9bc6-c07889622da0" containerName="maas-api"
Sep 07 11:48:36.997472 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:36.997319    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e9d1580-7203-4fbf-9f21-1cb58b51793e" containerName="maas-api"
Sep 07 11:48:36.997472 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:36.997327    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e9d1580-7203-4fbf-9f21-1cb58b51793e" containerName="maas-api"
Sep 07 11:48:36.997472 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:36.997416    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e9d1580-7203-4fbf-9f21-1cb58b51793e" containerName="maas-api"
Sep 07 11:48:36.997472 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:36.997443    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb6e6ff-fe80-4672-9bc6-c07889622da0" containerName="maas-api"
Sep 07 11:48:37.000660 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.000640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.002744 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.002719    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f866d4-serving-cert\""
Sep 07 11:48:37.010573 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.010544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"]
Sep 07 11:48:37.025451 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.025407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19b1b35-abbe-4f41-b429-5d29c9a65ada-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.025623 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.025508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-metrics-certs\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.025623 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.025560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljr9q\" (UniqueName: \"kubernetes.io/projected/c19b1b35-abbe-4f41-b429-5d29c9a65ada-kube-api-access-ljr9q\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.025758 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.025635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.126593 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.126550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljr9q\" (UniqueName: \"kubernetes.io/projected/c19b1b35-abbe-4f41-b429-5d29c9a65ada-kube-api-access-ljr9q\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.126828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.126614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.126828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.126653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19b1b35-abbe-4f41-b429-5d29c9a65ada-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.126828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.126691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-metrics-certs\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.126828 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:48:37.126717    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-serving-cert: secret "maas-api-e2e-ipp-b-f866d4-serving-cert" not found
Sep 07 11:48:37.126828 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:48:37.126820    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls podName:c19b1b35-abbe-4f41-b429-5d29c9a65ada nodeName:}" failed. No retries permitted until 2026-09-07 11:48:37.626775304 +0000 UTC m=+1567.332555975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls") pod "maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp" (UID: "c19b1b35-abbe-4f41-b429-5d29c9a65ada") : secret "maas-api-e2e-ipp-b-f866d4-serving-cert" not found
Sep 07 11:48:37.127378 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.127355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19b1b35-abbe-4f41-b429-5d29c9a65ada-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.129146 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.129122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-metrics-certs\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.135974 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.135952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljr9q\" (UniqueName: \"kubernetes.io/projected/c19b1b35-abbe-4f41-b429-5d29c9a65ada-kube-api-access-ljr9q\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.631216 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.631175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.633668 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.633636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:37.913075 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:37.912996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:38.187818 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:38.187770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"]
Sep 07 11:48:38.190411 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:48:38.190380    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19b1b35_abbe_4f41_b429_5d29c9a65ada.slice/crio-73cc97002cbbe65827f67d6f7ac70f813c98c513d8c11ec28a520172e87a495d WatchSource:0}: Error finding container 73cc97002cbbe65827f67d6f7ac70f813c98c513d8c11ec28a520172e87a495d: Status 404 returned error can't find the container with id 73cc97002cbbe65827f67d6f7ac70f813c98c513d8c11ec28a520172e87a495d
Sep 07 11:48:39.196731 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:39.196686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp" event={"ID":"c19b1b35-abbe-4f41-b429-5d29c9a65ada","Type":"ContainerStarted","Data":"6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147"}
Sep 07 11:48:39.197126 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:39.196757    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:39.197126 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:39.196779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp" event={"ID":"c19b1b35-abbe-4f41-b429-5d29c9a65ada","Type":"ContainerStarted","Data":"73cc97002cbbe65827f67d6f7ac70f813c98c513d8c11ec28a520172e87a495d"}
Sep 07 11:48:39.215115 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:39.215070    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp" podStartSLOduration=2.762176987 podStartE2EDuration="3.215055773s" totalImagesPullingTime="452.878786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:48:39.213459314 +0000 UTC m=+1568.919239996" watchObservedRunningTime="2026-09-07 11:48:39.215055773 +0000 UTC m=+1568.920836463"
Sep 07 11:48:44.283400 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.283366    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"]
Sep 07 11:48:44.283849 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.283759    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb" podUID="a39fb8e4-315a-4341-86a7-c7bafc2fe461" containerName="maas-api" containerID="cri-o://052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802" gracePeriod=30
Sep 07 11:48:44.537368 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.537310    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:48:44.595099 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.595067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-metrics-certs\") pod \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") "
Sep 07 11:48:44.595262 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.595112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39fb8e4-315a-4341-86a7-c7bafc2fe461-openshift-service-ca\") pod \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") "
Sep 07 11:48:44.595262 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.595172    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-maas-api-tls\") pod \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") "
Sep 07 11:48:44.595262 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.595232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm7vs\" (UniqueName: \"kubernetes.io/projected/a39fb8e4-315a-4341-86a7-c7bafc2fe461-kube-api-access-lm7vs\") pod \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\" (UID: \"a39fb8e4-315a-4341-86a7-c7bafc2fe461\") "
Sep 07 11:48:44.595526 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.595498    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39fb8e4-315a-4341-86a7-c7bafc2fe461-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a39fb8e4-315a-4341-86a7-c7bafc2fe461" (UID: "a39fb8e4-315a-4341-86a7-c7bafc2fe461"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:48:44.597217 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.597192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a39fb8e4-315a-4341-86a7-c7bafc2fe461" (UID: "a39fb8e4-315a-4341-86a7-c7bafc2fe461"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:48:44.597295 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.597220    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a39fb8e4-315a-4341-86a7-c7bafc2fe461" (UID: "a39fb8e4-315a-4341-86a7-c7bafc2fe461"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:48:44.597295 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.597277    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39fb8e4-315a-4341-86a7-c7bafc2fe461-kube-api-access-lm7vs" (OuterVolumeSpecName: "kube-api-access-lm7vs") pod "a39fb8e4-315a-4341-86a7-c7bafc2fe461" (UID: "a39fb8e4-315a-4341-86a7-c7bafc2fe461"). InnerVolumeSpecName "kube-api-access-lm7vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:48:44.697088 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.697052    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm7vs\" (UniqueName: \"kubernetes.io/projected/a39fb8e4-315a-4341-86a7-c7bafc2fe461-kube-api-access-lm7vs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:44.697088 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.697089    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:44.697386 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.697103    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39fb8e4-315a-4341-86a7-c7bafc2fe461-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:44.697386 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:44.697115    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39fb8e4-315a-4341-86a7-c7bafc2fe461-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:48:45.205761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.205686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:48:45.226484 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.226449    2581 generic.go:356] "Generic (PLEG): container finished" podID="a39fb8e4-315a-4341-86a7-c7bafc2fe461" containerID="052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802" exitCode=0
Sep 07 11:48:45.226622 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.226504    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"
Sep 07 11:48:45.226622 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.226506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb" event={"ID":"a39fb8e4-315a-4341-86a7-c7bafc2fe461","Type":"ContainerDied","Data":"052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802"}
Sep 07 11:48:45.226704 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.226628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb" event={"ID":"a39fb8e4-315a-4341-86a7-c7bafc2fe461","Type":"ContainerDied","Data":"b8e2a8c6f42fe4b658f8590e3c669605bda267eeefb639206e9a2bb00719aec6"}
Sep 07 11:48:45.226704 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.226652    2581 scope.go:117] "RemoveContainer" containerID="052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802"
Sep 07 11:48:45.240129 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.240034    2581 scope.go:117] "RemoveContainer" containerID="052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802"
Sep 07 11:48:45.242769 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:48:45.240639    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802\": container with ID starting with 052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802 not found: ID does not exist" containerID="052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802"
Sep 07 11:48:45.242769 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.240675    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802"} err="failed to get container status \"052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802\": rpc error: code = NotFound desc = could not find container \"052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802\": container with ID starting with 052cf1fd9dd89ae168bf7d0ee2ad6af78954a42e1d7557b526b7f05c4c1c6802 not found: ID does not exist"
Sep 07 11:48:45.251411 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.251346    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"]
Sep 07 11:48:45.258778 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:45.258750    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2195e0-6648c5d64b-6v2fb"]
Sep 07 11:48:46.942496 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:48:46.942448    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a39fb8e4-315a-4341-86a7-c7bafc2fe461" path="/var/lib/kubelet/pods/a39fb8e4-315a-4341-86a7-c7bafc2fe461/volumes"
Sep 07 11:49:39.607592 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.607558    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"]
Sep 07 11:49:39.608188 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.608087    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39fb8e4-315a-4341-86a7-c7bafc2fe461" containerName="maas-api"
Sep 07 11:49:39.608188 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.608108    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39fb8e4-315a-4341-86a7-c7bafc2fe461" containerName="maas-api"
Sep 07 11:49:39.608310 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.608231    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a39fb8e4-315a-4341-86a7-c7bafc2fe461" containerName="maas-api"
Sep 07 11:49:39.611454 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.611428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.613646 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.613620    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-55c3cf-serving-cert\""
Sep 07 11:49:39.621889 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.621865    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"]
Sep 07 11:49:39.693265 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.693232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.693444 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.693302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cbc\" (UniqueName: \"kubernetes.io/projected/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-kube-api-access-w7cbc\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.693444 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.693348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.693444 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.693402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.794174 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.794140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.794340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.794184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.794340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.794236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cbc\" (UniqueName: \"kubernetes.io/projected/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-kube-api-access-w7cbc\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.794340 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.794263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.794465 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:49:39.794363    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-serving-cert: secret "maas-api-e2e-ipp-cleanup-55c3cf-serving-cert" not found
Sep 07 11:49:39.794465 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:49:39.794416    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls podName:d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb nodeName:}" failed. No retries permitted until 2026-09-07 11:49:40.294401314 +0000 UTC m=+1630.000181996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd" (UID: "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb") : secret "maas-api-e2e-ipp-cleanup-55c3cf-serving-cert" not found
Sep 07 11:49:39.794902 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.794872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.796558 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.796538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:39.804359 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:39.804339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cbc\" (UniqueName: \"kubernetes.io/projected/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-kube-api-access-w7cbc\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:40.299766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:40.299732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:40.302152 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:40.302130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:40.524339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:40.524301    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:40.798451 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:40.798425    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"]
Sep 07 11:49:40.801279 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:49:40.801253    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36e0ba2_9a92_484c_9ee5_0b1abaf5adcb.slice/crio-d2fd86b89d9d5e09704e99458c95136d44f86aca852bcf93ce7f008182791bab WatchSource:0}: Error finding container d2fd86b89d9d5e09704e99458c95136d44f86aca852bcf93ce7f008182791bab: Status 404 returned error can't find the container with id d2fd86b89d9d5e09704e99458c95136d44f86aca852bcf93ce7f008182791bab
Sep 07 11:49:41.517469 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:41.516699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd" event={"ID":"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb","Type":"ContainerStarted","Data":"13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282"}
Sep 07 11:49:41.517469 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:41.516822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd" event={"ID":"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb","Type":"ContainerStarted","Data":"d2fd86b89d9d5e09704e99458c95136d44f86aca852bcf93ce7f008182791bab"}
Sep 07 11:49:41.517469 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:41.517058    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:49:41.535498 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:41.535449    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd" podStartSLOduration=2.170355011 podStartE2EDuration="2.535435252s" totalImagesPullingTime="365.080241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:49:41.535011916 +0000 UTC m=+1631.240792607" watchObservedRunningTime="2026-09-07 11:49:41.535435252 +0000 UTC m=+1631.241215942"
Sep 07 11:49:47.528348 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:49:47.528317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:50:10.011517 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.011481    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"]
Sep 07 11:50:10.012034 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.011716    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd" podUID="d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" containerName="maas-api" containerID="cri-o://13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282" gracePeriod=30
Sep 07 11:50:10.263006 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.262945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:50:10.369160 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.369114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls\") pod \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") "
Sep 07 11:50:10.369339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.369186    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7cbc\" (UniqueName: \"kubernetes.io/projected/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-kube-api-access-w7cbc\") pod \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") "
Sep 07 11:50:10.369339 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.369288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-metrics-certs\") pod \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") "
Sep 07 11:50:10.369534 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.369382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-openshift-service-ca\") pod \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\" (UID: \"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb\") "
Sep 07 11:50:10.369735 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.369704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" (UID: "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:50:10.371302 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.371271    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" (UID: "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:50:10.371302 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.371284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" (UID: "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:50:10.371451 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.371389    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-kube-api-access-w7cbc" (OuterVolumeSpecName: "kube-api-access-w7cbc") pod "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" (UID: "d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb"). InnerVolumeSpecName "kube-api-access-w7cbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:50:10.471125 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.471097    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:50:10.471125 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.471121    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7cbc\" (UniqueName: \"kubernetes.io/projected/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-kube-api-access-w7cbc\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:50:10.471125 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.471130    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:50:10.471341 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.471139    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:50:10.674621 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.674532    2581 generic.go:356] "Generic (PLEG): container finished" podID="d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" containerID="13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282" exitCode=0
Sep 07 11:50:10.674621 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.674573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"
Sep 07 11:50:10.674819 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.674624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd" event={"ID":"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb","Type":"ContainerDied","Data":"13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282"}
Sep 07 11:50:10.674819 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.674676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd" event={"ID":"d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb","Type":"ContainerDied","Data":"d2fd86b89d9d5e09704e99458c95136d44f86aca852bcf93ce7f008182791bab"}
Sep 07 11:50:10.674819 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.674699    2581 scope.go:117] "RemoveContainer" containerID="13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282"
Sep 07 11:50:10.683945 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.683922    2581 scope.go:117] "RemoveContainer" containerID="13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282"
Sep 07 11:50:10.684213 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:50:10.684195    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282\": container with ID starting with 13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282 not found: ID does not exist" containerID="13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282"
Sep 07 11:50:10.684288 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.684219    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282"} err="failed to get container status \"13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282\": rpc error: code = NotFound desc = could not find container \"13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282\": container with ID starting with 13f7b2d346f8369516efc0bce48cbaafbfad178a031193d0b1e0a3879c522282 not found: ID does not exist"
Sep 07 11:50:10.696207 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.695587    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"]
Sep 07 11:50:10.697637 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.697619    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-55c3cf-5b77499569-kbqxd"]
Sep 07 11:50:10.942990 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:10.942903    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" path="/var/lib/kubelet/pods/d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb/volumes"
Sep 07 11:50:36.843732 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:36.843695    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"]
Sep 07 11:50:36.844554 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:36.844522    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz" podUID="b66406c3-ffc7-4b55-a92d-acd69954cbdf" containerName="maas-api" containerID="cri-o://abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e" gracePeriod=30
Sep 07 11:50:37.095678 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.095620    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:50:37.206540 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.206509    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls\") pod \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") "
Sep 07 11:50:37.206711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.206556    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-metrics-certs\") pod \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") "
Sep 07 11:50:37.206711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.206628    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66406c3-ffc7-4b55-a92d-acd69954cbdf-openshift-service-ca\") pod \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") "
Sep 07 11:50:37.206711 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.206677    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8l6t\" (UniqueName: \"kubernetes.io/projected/b66406c3-ffc7-4b55-a92d-acd69954cbdf-kube-api-access-t8l6t\") pod \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\" (UID: \"b66406c3-ffc7-4b55-a92d-acd69954cbdf\") "
Sep 07 11:50:37.207098 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.207070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66406c3-ffc7-4b55-a92d-acd69954cbdf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b66406c3-ffc7-4b55-a92d-acd69954cbdf" (UID: "b66406c3-ffc7-4b55-a92d-acd69954cbdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:50:37.208623 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.208588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b66406c3-ffc7-4b55-a92d-acd69954cbdf" (UID: "b66406c3-ffc7-4b55-a92d-acd69954cbdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:50:37.208714 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.208643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b66406c3-ffc7-4b55-a92d-acd69954cbdf" (UID: "b66406c3-ffc7-4b55-a92d-acd69954cbdf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:50:37.208714 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.208694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66406c3-ffc7-4b55-a92d-acd69954cbdf-kube-api-access-t8l6t" (OuterVolumeSpecName: "kube-api-access-t8l6t") pod "b66406c3-ffc7-4b55-a92d-acd69954cbdf" (UID: "b66406c3-ffc7-4b55-a92d-acd69954cbdf"). InnerVolumeSpecName "kube-api-access-t8l6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:50:37.307766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.307735    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8l6t\" (UniqueName: \"kubernetes.io/projected/b66406c3-ffc7-4b55-a92d-acd69954cbdf-kube-api-access-t8l6t\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:50:37.307766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.307759    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:50:37.307766 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.307769    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66406c3-ffc7-4b55-a92d-acd69954cbdf-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:50:37.307991 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.307777    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66406c3-ffc7-4b55-a92d-acd69954cbdf-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:50:37.806849 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.806813    2581 generic.go:356] "Generic (PLEG): container finished" podID="b66406c3-ffc7-4b55-a92d-acd69954cbdf" containerID="abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e" exitCode=0
Sep 07 11:50:37.807009 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.806861    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"
Sep 07 11:50:37.807009 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.806898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz" event={"ID":"b66406c3-ffc7-4b55-a92d-acd69954cbdf","Type":"ContainerDied","Data":"abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e"}
Sep 07 11:50:37.807009 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.806962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz" event={"ID":"b66406c3-ffc7-4b55-a92d-acd69954cbdf","Type":"ContainerDied","Data":"9e4cd26dda16624f35ab51c45bf560b9a672ef66a50c3cc6a17b67fbea734927"}
Sep 07 11:50:37.807009 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.806994    2581 scope.go:117] "RemoveContainer" containerID="abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e"
Sep 07 11:50:37.816924 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.816904    2581 scope.go:117] "RemoveContainer" containerID="abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e"
Sep 07 11:50:37.817234 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:50:37.817212    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e\": container with ID starting with abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e not found: ID does not exist" containerID="abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e"
Sep 07 11:50:37.817336 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.817241    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e"} err="failed to get container status \"abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e\": rpc error: code = NotFound desc = could not find container \"abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e\": container with ID starting with abe905fd2efb031d3e4f9c5b298290fd69b9927682f031775b81d0e17fd07a0e not found: ID does not exist"
Sep 07 11:50:37.829237 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.829212    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"]
Sep 07 11:50:37.832664 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:37.832642    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-984178-85dd4bc6c4-c4qsz"]
Sep 07 11:50:38.943311 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:50:38.943276    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b66406c3-ffc7-4b55-a92d-acd69954cbdf" path="/var/lib/kubelet/pods/b66406c3-ffc7-4b55-a92d-acd69954cbdf/volumes"
Sep 07 11:51:37.201906 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.201865    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"]
Sep 07 11:51:37.202416 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.202078    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp" podUID="c19b1b35-abbe-4f41-b429-5d29c9a65ada" containerName="maas-api" containerID="cri-o://6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147" gracePeriod=30
Sep 07 11:51:37.447881 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.447854    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:51:37.545470 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.545433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls\") pod \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") "
Sep 07 11:51:37.545619 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.545547    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-metrics-certs\") pod \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") "
Sep 07 11:51:37.545619 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.545599    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19b1b35-abbe-4f41-b429-5d29c9a65ada-openshift-service-ca\") pod \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") "
Sep 07 11:51:37.545692 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.545655    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljr9q\" (UniqueName: \"kubernetes.io/projected/c19b1b35-abbe-4f41-b429-5d29c9a65ada-kube-api-access-ljr9q\") pod \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\" (UID: \"c19b1b35-abbe-4f41-b429-5d29c9a65ada\") "
Sep 07 11:51:37.546007 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.545978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19b1b35-abbe-4f41-b429-5d29c9a65ada-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c19b1b35-abbe-4f41-b429-5d29c9a65ada" (UID: "c19b1b35-abbe-4f41-b429-5d29c9a65ada"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:51:37.547493 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.547472    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c19b1b35-abbe-4f41-b429-5d29c9a65ada" (UID: "c19b1b35-abbe-4f41-b429-5d29c9a65ada"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:51:37.547717 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.547697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c19b1b35-abbe-4f41-b429-5d29c9a65ada" (UID: "c19b1b35-abbe-4f41-b429-5d29c9a65ada"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:51:37.548007 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.547983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19b1b35-abbe-4f41-b429-5d29c9a65ada-kube-api-access-ljr9q" (OuterVolumeSpecName: "kube-api-access-ljr9q") pod "c19b1b35-abbe-4f41-b429-5d29c9a65ada" (UID: "c19b1b35-abbe-4f41-b429-5d29c9a65ada"). InnerVolumeSpecName "kube-api-access-ljr9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:51:37.646445 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.646408    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19b1b35-abbe-4f41-b429-5d29c9a65ada-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:51:37.646445 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.646439    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljr9q\" (UniqueName: \"kubernetes.io/projected/c19b1b35-abbe-4f41-b429-5d29c9a65ada-kube-api-access-ljr9q\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:51:37.646445 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.646449    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:51:37.646669 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:37.646458    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19b1b35-abbe-4f41-b429-5d29c9a65ada-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:51:38.103159 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.103125    2581 generic.go:356] "Generic (PLEG): container finished" podID="c19b1b35-abbe-4f41-b429-5d29c9a65ada" containerID="6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147" exitCode=0
Sep 07 11:51:38.103318 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.103165    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"
Sep 07 11:51:38.103318 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.103206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp" event={"ID":"c19b1b35-abbe-4f41-b429-5d29c9a65ada","Type":"ContainerDied","Data":"6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147"}
Sep 07 11:51:38.103318 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.103254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp" event={"ID":"c19b1b35-abbe-4f41-b429-5d29c9a65ada","Type":"ContainerDied","Data":"73cc97002cbbe65827f67d6f7ac70f813c98c513d8c11ec28a520172e87a495d"}
Sep 07 11:51:38.103318 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.103278    2581 scope.go:117] "RemoveContainer" containerID="6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147"
Sep 07 11:51:38.111827 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.111783    2581 scope.go:117] "RemoveContainer" containerID="6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147"
Sep 07 11:51:38.112083 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:51:38.112064    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147\": container with ID starting with 6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147 not found: ID does not exist" containerID="6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147"
Sep 07 11:51:38.112152 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.112096    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147"} err="failed to get container status \"6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147\": rpc error: code = NotFound desc = could not find container \"6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147\": container with ID starting with 6940460d0d21d3a73b3873c0191a82c5d475a1b0df0d65115ef22c15075f6147 not found: ID does not exist"
Sep 07 11:51:38.124498 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.124476    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"]
Sep 07 11:51:38.128094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.128075    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f866d4-6d4bc8d8f8-c6qnp"]
Sep 07 11:51:38.941439 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:38.941403    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c19b1b35-abbe-4f41-b429-5d29c9a65ada" path="/var/lib/kubelet/pods/c19b1b35-abbe-4f41-b429-5d29c9a65ada/volumes"
Sep 07 11:51:55.056275 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.056242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"]
Sep 07 11:51:55.056773 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.056502    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8" podUID="b83e096d-6811-4235-8ee7-af0a54e02316" containerName="maas-api" containerID="cri-o://d58684a4468a1720c294d9807be3a69f030aff26e127c72230209f0921bf2679" gracePeriod=30
Sep 07 11:51:55.199913 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.199872    2581 generic.go:356] "Generic (PLEG): container finished" podID="b83e096d-6811-4235-8ee7-af0a54e02316" containerID="d58684a4468a1720c294d9807be3a69f030aff26e127c72230209f0921bf2679" exitCode=0
Sep 07 11:51:55.200107 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.199947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8" event={"ID":"b83e096d-6811-4235-8ee7-af0a54e02316","Type":"ContainerDied","Data":"d58684a4468a1720c294d9807be3a69f030aff26e127c72230209f0921bf2679"}
Sep 07 11:51:55.312311 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.312252    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:51:55.400010 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.399980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-metrics-certs\") pod \"b83e096d-6811-4235-8ee7-af0a54e02316\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") "
Sep 07 11:51:55.400183 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.400028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-maas-api-tls\") pod \"b83e096d-6811-4235-8ee7-af0a54e02316\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") "
Sep 07 11:51:55.400183 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.400091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83e096d-6811-4235-8ee7-af0a54e02316-openshift-service-ca\") pod \"b83e096d-6811-4235-8ee7-af0a54e02316\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") "
Sep 07 11:51:55.400183 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.400115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbk9f\" (UniqueName: \"kubernetes.io/projected/b83e096d-6811-4235-8ee7-af0a54e02316-kube-api-access-pbk9f\") pod \"b83e096d-6811-4235-8ee7-af0a54e02316\" (UID: \"b83e096d-6811-4235-8ee7-af0a54e02316\") "
Sep 07 11:51:55.400458 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.400431    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83e096d-6811-4235-8ee7-af0a54e02316-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b83e096d-6811-4235-8ee7-af0a54e02316" (UID: "b83e096d-6811-4235-8ee7-af0a54e02316"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:51:55.402094 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.402068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b83e096d-6811-4235-8ee7-af0a54e02316" (UID: "b83e096d-6811-4235-8ee7-af0a54e02316"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:51:55.402188 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.402115    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b83e096d-6811-4235-8ee7-af0a54e02316" (UID: "b83e096d-6811-4235-8ee7-af0a54e02316"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:51:55.402237 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.402206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83e096d-6811-4235-8ee7-af0a54e02316-kube-api-access-pbk9f" (OuterVolumeSpecName: "kube-api-access-pbk9f") pod "b83e096d-6811-4235-8ee7-af0a54e02316" (UID: "b83e096d-6811-4235-8ee7-af0a54e02316"). InnerVolumeSpecName "kube-api-access-pbk9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:51:55.501628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.501596    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:51:55.501628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.501620    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83e096d-6811-4235-8ee7-af0a54e02316-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:51:55.501628 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.501629    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83e096d-6811-4235-8ee7-af0a54e02316-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:51:55.501891 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:55.501641    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbk9f\" (UniqueName: \"kubernetes.io/projected/b83e096d-6811-4235-8ee7-af0a54e02316-kube-api-access-pbk9f\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:51:56.205914 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:56.205885    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"
Sep 07 11:51:56.206413 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:56.205937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8" event={"ID":"b83e096d-6811-4235-8ee7-af0a54e02316","Type":"ContainerDied","Data":"641bbb019cf813e3dec2b26765a51479bab2c9f8b0de0742c551f39396a76cf5"}
Sep 07 11:51:56.206413 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:56.206001    2581 scope.go:117] "RemoveContainer" containerID="d58684a4468a1720c294d9807be3a69f030aff26e127c72230209f0921bf2679"
Sep 07 11:51:56.237491 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:56.237459    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"]
Sep 07 11:51:56.244155 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:56.244134    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b944b-58f565f9df-66xx8"]
Sep 07 11:51:56.941679 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:51:56.941646    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b83e096d-6811-4235-8ee7-af0a54e02316" path="/var/lib/kubelet/pods/b83e096d-6811-4235-8ee7-af0a54e02316/volumes"
Sep 07 11:52:13.120886 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.120601    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"]
Sep 07 11:52:13.121881 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.121844    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5" podUID="edeba831-43ba-4e67-bf30-07adb4a3b224" containerName="maas-api" containerID="cri-o://9debe618dab8fb986214bd64111b6063136b7ed7d4123428517d35d1228f80d5" gracePeriod=30
Sep 07 11:52:13.295709 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.295679    2581 generic.go:356] "Generic (PLEG): container finished" podID="edeba831-43ba-4e67-bf30-07adb4a3b224" containerID="9debe618dab8fb986214bd64111b6063136b7ed7d4123428517d35d1228f80d5" exitCode=0
Sep 07 11:52:13.295862 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.295733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5" event={"ID":"edeba831-43ba-4e67-bf30-07adb4a3b224","Type":"ContainerDied","Data":"9debe618dab8fb986214bd64111b6063136b7ed7d4123428517d35d1228f80d5"}
Sep 07 11:52:13.370196 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.370173    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:52:13.455644 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.455566    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeba831-43ba-4e67-bf30-07adb4a3b224-openshift-service-ca\") pod \"edeba831-43ba-4e67-bf30-07adb4a3b224\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") "
Sep 07 11:52:13.455644 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.455630    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq4k9\" (UniqueName: \"kubernetes.io/projected/edeba831-43ba-4e67-bf30-07adb4a3b224-kube-api-access-bq4k9\") pod \"edeba831-43ba-4e67-bf30-07adb4a3b224\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") "
Sep 07 11:52:13.455841 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.455684    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-metrics-certs\") pod \"edeba831-43ba-4e67-bf30-07adb4a3b224\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") "
Sep 07 11:52:13.455841 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.455722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-maas-api-tls\") pod \"edeba831-43ba-4e67-bf30-07adb4a3b224\" (UID: \"edeba831-43ba-4e67-bf30-07adb4a3b224\") "
Sep 07 11:52:13.456589 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.456555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edeba831-43ba-4e67-bf30-07adb4a3b224-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edeba831-43ba-4e67-bf30-07adb4a3b224" (UID: "edeba831-43ba-4e67-bf30-07adb4a3b224"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:52:13.458084 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.458055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edeba831-43ba-4e67-bf30-07adb4a3b224-kube-api-access-bq4k9" (OuterVolumeSpecName: "kube-api-access-bq4k9") pod "edeba831-43ba-4e67-bf30-07adb4a3b224" (UID: "edeba831-43ba-4e67-bf30-07adb4a3b224"). InnerVolumeSpecName "kube-api-access-bq4k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:52:13.458318 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.458299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "edeba831-43ba-4e67-bf30-07adb4a3b224" (UID: "edeba831-43ba-4e67-bf30-07adb4a3b224"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:52:13.458371 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.458351    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edeba831-43ba-4e67-bf30-07adb4a3b224" (UID: "edeba831-43ba-4e67-bf30-07adb4a3b224"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:52:13.556677 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.556646    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:52:13.556677 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.556674    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edeba831-43ba-4e67-bf30-07adb4a3b224-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:52:13.556677 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.556683    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeba831-43ba-4e67-bf30-07adb4a3b224-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:52:13.556921 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:13.556692    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq4k9\" (UniqueName: \"kubernetes.io/projected/edeba831-43ba-4e67-bf30-07adb4a3b224-kube-api-access-bq4k9\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:52:14.301520 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:14.301491    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"
Sep 07 11:52:14.301952 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:14.301525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5" event={"ID":"edeba831-43ba-4e67-bf30-07adb4a3b224","Type":"ContainerDied","Data":"3a514146c5a723a75b61bf51624a831008e16bdf9d769ecacdb010ac6b368e70"}
Sep 07 11:52:14.301952 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:14.301582    2581 scope.go:117] "RemoveContainer" containerID="9debe618dab8fb986214bd64111b6063136b7ed7d4123428517d35d1228f80d5"
Sep 07 11:52:14.322553 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:14.322527    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"]
Sep 07 11:52:14.326183 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:14.326162    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f0a2f-596b5cdd44-7n2z5"]
Sep 07 11:52:14.941317 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:14.941279    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edeba831-43ba-4e67-bf30-07adb4a3b224" path="/var/lib/kubelet/pods/edeba831-43ba-4e67-bf30-07adb4a3b224/volumes"
Sep 07 11:52:23.113245 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.113160    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f74dc8dd5-gpqwd"]
Sep 07 11:52:23.113747 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.113458    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd" podUID="05b120ff-2097-407b-a942-106d280f99cb" containerName="manager" containerID="cri-o://502229a439276a07a905016a60b99ccb3a42acf6be017099fe83a5463fa98363" gracePeriod=30
Sep 07 11:52:23.347456 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.347421    2581 generic.go:356] "Generic (PLEG): container finished" podID="05b120ff-2097-407b-a942-106d280f99cb" containerID="502229a439276a07a905016a60b99ccb3a42acf6be017099fe83a5463fa98363" exitCode=0
Sep 07 11:52:23.347631 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.347514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd" event={"ID":"05b120ff-2097-407b-a942-106d280f99cb","Type":"ContainerDied","Data":"502229a439276a07a905016a60b99ccb3a42acf6be017099fe83a5463fa98363"}
Sep 07 11:52:23.370386 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.370335    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:52:23.438050 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.438021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-metrics-certs\") pod \"05b120ff-2097-407b-a942-106d280f99cb\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") "
Sep 07 11:52:23.438241 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.438131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wkvx\" (UniqueName: \"kubernetes.io/projected/05b120ff-2097-407b-a942-106d280f99cb-kube-api-access-9wkvx\") pod \"05b120ff-2097-407b-a942-106d280f99cb\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") "
Sep 07 11:52:23.438241 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.438189    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-webhook-cert\") pod \"05b120ff-2097-407b-a942-106d280f99cb\" (UID: \"05b120ff-2097-407b-a942-106d280f99cb\") "
Sep 07 11:52:23.440375 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.440344    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "05b120ff-2097-407b-a942-106d280f99cb" (UID: "05b120ff-2097-407b-a942-106d280f99cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:52:23.440621 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.440585    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b120ff-2097-407b-a942-106d280f99cb-kube-api-access-9wkvx" (OuterVolumeSpecName: "kube-api-access-9wkvx") pod "05b120ff-2097-407b-a942-106d280f99cb" (UID: "05b120ff-2097-407b-a942-106d280f99cb"). InnerVolumeSpecName "kube-api-access-9wkvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:52:23.440621 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.440599    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "05b120ff-2097-407b-a942-106d280f99cb" (UID: "05b120ff-2097-407b-a942-106d280f99cb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:52:23.538965 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.538932    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wkvx\" (UniqueName: \"kubernetes.io/projected/05b120ff-2097-407b-a942-106d280f99cb-kube-api-access-9wkvx\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:52:23.538965 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.538959    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-webhook-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:52:23.538965 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:23.538969    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b120ff-2097-407b-a942-106d280f99cb-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:52:24.353827 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:24.353771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd" event={"ID":"05b120ff-2097-407b-a942-106d280f99cb","Type":"ContainerDied","Data":"68fda04ec833c2d7e21ce1f6b73d77ee10654cccb8eb5d2273d16d6b58ec7f1b"}
Sep 07 11:52:24.353827 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:24.353817    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f74dc8dd5-gpqwd"
Sep 07 11:52:24.354366 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:24.353845    2581 scope.go:117] "RemoveContainer" containerID="502229a439276a07a905016a60b99ccb3a42acf6be017099fe83a5463fa98363"
Sep 07 11:52:24.375951 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:24.375923    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f74dc8dd5-gpqwd"]
Sep 07 11:52:24.380404 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:24.380380    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f74dc8dd5-gpqwd"]
Sep 07 11:52:24.940438 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:52:24.940401    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05b120ff-2097-407b-a942-106d280f99cb" path="/var/lib/kubelet/pods/05b120ff-2097-407b-a942-106d280f99cb/volumes"
Sep 07 11:55:03.129919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:03.129885    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"]
Sep 07 11:55:03.130437 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:03.130069    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl" podUID="cc6a6c66-86c6-4263-93bf-f73258c01180" containerName="manager" containerID="cri-o://97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437" gracePeriod=10
Sep 07 11:55:05.768332 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:05.768303    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:55:05.925019 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:05.924925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcvf5\" (UniqueName: \"kubernetes.io/projected/cc6a6c66-86c6-4263-93bf-f73258c01180-kube-api-access-tcvf5\") pod \"cc6a6c66-86c6-4263-93bf-f73258c01180\" (UID: \"cc6a6c66-86c6-4263-93bf-f73258c01180\") "
Sep 07 11:55:05.925019 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:05.924999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc6a6c66-86c6-4263-93bf-f73258c01180-extensions-socket-volume\") pod \"cc6a6c66-86c6-4263-93bf-f73258c01180\" (UID: \"cc6a6c66-86c6-4263-93bf-f73258c01180\") "
Sep 07 11:55:05.925601 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:05.925575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6a6c66-86c6-4263-93bf-f73258c01180-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cc6a6c66-86c6-4263-93bf-f73258c01180" (UID: "cc6a6c66-86c6-4263-93bf-f73258c01180"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:55:05.929979 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:05.929953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6a6c66-86c6-4263-93bf-f73258c01180-kube-api-access-tcvf5" (OuterVolumeSpecName: "kube-api-access-tcvf5") pod "cc6a6c66-86c6-4263-93bf-f73258c01180" (UID: "cc6a6c66-86c6-4263-93bf-f73258c01180"). InnerVolumeSpecName "kube-api-access-tcvf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:55:06.025807 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.025767    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcvf5\" (UniqueName: \"kubernetes.io/projected/cc6a6c66-86c6-4263-93bf-f73258c01180-kube-api-access-tcvf5\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:55:06.025959 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.025814    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc6a6c66-86c6-4263-93bf-f73258c01180-extensions-socket-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:55:06.100837 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.100772    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc6a6c66-86c6-4263-93bf-f73258c01180" containerID="97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437" exitCode=0
Sep 07 11:55:06.101003 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.100836    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"
Sep 07 11:55:06.101003 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.100860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl" event={"ID":"cc6a6c66-86c6-4263-93bf-f73258c01180","Type":"ContainerDied","Data":"97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437"}
Sep 07 11:55:06.101003 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.100932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl" event={"ID":"cc6a6c66-86c6-4263-93bf-f73258c01180","Type":"ContainerDied","Data":"ca9cd840587b59831d6c82ff8ab9e0259729f72df380c362a4188ebecdaa271a"}
Sep 07 11:55:06.101003 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.100965    2581 scope.go:117] "RemoveContainer" containerID="97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437"
Sep 07 11:55:06.112477 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.112461    2581 scope.go:117] "RemoveContainer" containerID="97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437"
Sep 07 11:55:06.112746 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:55:06.112729    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437\": container with ID starting with 97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437 not found: ID does not exist" containerID="97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437"
Sep 07 11:55:06.112828 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.112752    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437"} err="failed to get container status \"97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437\": rpc error: code = NotFound desc = could not find container \"97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437\": container with ID starting with 97b5abe7dc2ccc232cae5eac5cd8b264610c1e6faf7473a4e0d3f9c933b88437 not found: ID does not exist"
Sep 07 11:55:06.122989 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.122969    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"]
Sep 07 11:55:06.127990 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.127969    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-mb6pl"]
Sep 07 11:55:06.940221 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:55:06.940189    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6a6c66-86c6-4263-93bf-f73258c01180" path="/var/lib/kubelet/pods/cc6a6c66-86c6-4263-93bf-f73258c01180/volumes"
Sep 07 11:56:03.881860 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.881821    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"]
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882156    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6a6c66-86c6-4263-93bf-f73258c01180" containerName="manager"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882167    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6a6c66-86c6-4263-93bf-f73258c01180" containerName="manager"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882177    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b120ff-2097-407b-a942-106d280f99cb" containerName="manager"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882182    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b120ff-2097-407b-a942-106d280f99cb" containerName="manager"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882195    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b66406c3-ffc7-4b55-a92d-acd69954cbdf" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882201    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b66406c3-ffc7-4b55-a92d-acd69954cbdf" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882211    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83e096d-6811-4235-8ee7-af0a54e02316" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882216    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83e096d-6811-4235-8ee7-af0a54e02316" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882223    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882228    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882237    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edeba831-43ba-4e67-bf30-07adb4a3b224" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882242    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edeba831-43ba-4e67-bf30-07adb4a3b224" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882250    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19b1b35-abbe-4f41-b429-5d29c9a65ada" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882254    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19b1b35-abbe-4f41-b429-5d29c9a65ada" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882300    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="05b120ff-2097-407b-a942-106d280f99cb" containerName="manager"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882307    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b66406c3-ffc7-4b55-a92d-acd69954cbdf" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882313    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36e0ba2-9a92-484c-9ee5-0b1abaf5adcb" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882321    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6a6c66-86c6-4263-93bf-f73258c01180" containerName="manager"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882326    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19b1b35-abbe-4f41-b429-5d29c9a65ada" containerName="maas-api"
Sep 07 11:56:03.882323 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882331    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83e096d-6811-4235-8ee7-af0a54e02316" containerName="maas-api"
Sep 07 11:56:03.882939 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.882337    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edeba831-43ba-4e67-bf30-07adb4a3b224" containerName="maas-api"
Sep 07 11:56:03.884894 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.884878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:03.887379 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.887357    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6brmr\""
Sep 07 11:56:03.899460 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.899435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"]
Sep 07 11:56:03.997429 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.997381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8p9b\" (UniqueName: \"kubernetes.io/projected/77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb-kube-api-access-w8p9b\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-g7xhf\" (UID: \"77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:03.997633 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:03.997523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-g7xhf\" (UID: \"77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:04.098076 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:04.098035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-g7xhf\" (UID: \"77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:04.098076 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:04.098081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8p9b\" (UniqueName: \"kubernetes.io/projected/77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb-kube-api-access-w8p9b\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-g7xhf\" (UID: \"77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:04.098527 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:04.098498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-g7xhf\" (UID: \"77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:04.105804 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:04.105765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8p9b\" (UniqueName: \"kubernetes.io/projected/77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb-kube-api-access-w8p9b\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-g7xhf\" (UID: \"77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:04.194886 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:04.194769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:04.470506 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:04.470477    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"]
Sep 07 11:56:04.473891 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:56:04.473863    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fbec0c_5f4e_434f_8fa3_a4e0d641f9cb.slice/crio-3324df66de1a1b333b6d2425e491376ddfcbbd26b59c70281181935fb982d424 WatchSource:0}: Error finding container 3324df66de1a1b333b6d2425e491376ddfcbbd26b59c70281181935fb982d424: Status 404 returned error can't find the container with id 3324df66de1a1b333b6d2425e491376ddfcbbd26b59c70281181935fb982d424
Sep 07 11:56:05.364899 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:05.364859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf" event={"ID":"77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb","Type":"ContainerStarted","Data":"87def5f2ff2ff7717416bbcfe57b3e11e30837b2441e613485b1d9bf3ef88940"}
Sep 07 11:56:05.365381 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:05.364952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf" event={"ID":"77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb","Type":"ContainerStarted","Data":"3324df66de1a1b333b6d2425e491376ddfcbbd26b59c70281181935fb982d424"}
Sep 07 11:56:05.365381 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:05.364997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:56:05.388288 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:05.388205    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf" podStartSLOduration=2.388186256 podStartE2EDuration="2.388186256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:56:05.386976623 +0000 UTC m=+2015.092757312" watchObservedRunningTime="2026-09-07 11:56:05.388186256 +0000 UTC m=+2015.093966947"
Sep 07 11:56:16.371572 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:56:16.371533    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-g7xhf"
Sep 07 11:58:05.770298 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.770261    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"]
Sep 07 11:58:05.773657 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.773637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.783475 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.783451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"]
Sep 07 11:58:05.842092 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.842055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e516238d-eb33-4138-b6b1-b2075096b1e5-maas-api-tls\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.842269 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.842106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wht6\" (UniqueName: \"kubernetes.io/projected/e516238d-eb33-4138-b6b1-b2075096b1e5-kube-api-access-7wht6\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.842269 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.842188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e516238d-eb33-4138-b6b1-b2075096b1e5-metrics-certs\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.842269 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.842230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e516238d-eb33-4138-b6b1-b2075096b1e5-openshift-service-ca\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.942767 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.942732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wht6\" (UniqueName: \"kubernetes.io/projected/e516238d-eb33-4138-b6b1-b2075096b1e5-kube-api-access-7wht6\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.942767 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.942773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e516238d-eb33-4138-b6b1-b2075096b1e5-metrics-certs\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.943031 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.942802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e516238d-eb33-4138-b6b1-b2075096b1e5-openshift-service-ca\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.943031 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.942866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e516238d-eb33-4138-b6b1-b2075096b1e5-maas-api-tls\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.943554 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.943530    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e516238d-eb33-4138-b6b1-b2075096b1e5-openshift-service-ca\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.945158 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.945138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e516238d-eb33-4138-b6b1-b2075096b1e5-metrics-certs\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.945205 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.945181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e516238d-eb33-4138-b6b1-b2075096b1e5-maas-api-tls\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:05.950779 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:05.950758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wht6\" (UniqueName: \"kubernetes.io/projected/e516238d-eb33-4138-b6b1-b2075096b1e5-kube-api-access-7wht6\") pod \"maas-api-75d79b99ff-xg24v\" (UID: \"e516238d-eb33-4138-b6b1-b2075096b1e5\") " pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:06.085255 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:06.085173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:06.358648 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:06.358622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"]
Sep 07 11:58:06.360740 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:58:06.360714    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode516238d_eb33_4138_b6b1_b2075096b1e5.slice/crio-902e09ef1b6205e32f169dc0ac93ecc5faaba16092dfb5fe133c9fab38e85061 WatchSource:0}: Error finding container 902e09ef1b6205e32f169dc0ac93ecc5faaba16092dfb5fe133c9fab38e85061: Status 404 returned error can't find the container with id 902e09ef1b6205e32f169dc0ac93ecc5faaba16092dfb5fe133c9fab38e85061
Sep 07 11:58:06.362080 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:06.362062    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 11:58:06.921045 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:06.921010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v" event={"ID":"e516238d-eb33-4138-b6b1-b2075096b1e5","Type":"ContainerStarted","Data":"902e09ef1b6205e32f169dc0ac93ecc5faaba16092dfb5fe133c9fab38e85061"}
Sep 07 11:58:07.927013 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:07.926924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v" event={"ID":"e516238d-eb33-4138-b6b1-b2075096b1e5","Type":"ContainerStarted","Data":"0e0f7a361fb194aa6ad3cbbdc1833029c08846d98079f6ad7791a6c4ca7c0327"}
Sep 07 11:58:07.927013 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:07.927004    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:07.944962 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:07.944906    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v" podStartSLOduration=1.839075198 podStartE2EDuration="2.944886563s" totalImagesPullingTime="1.105811365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:58:07.94432389 +0000 UTC m=+2137.650104604" watchObservedRunningTime="2026-09-07 11:58:07.944886563 +0000 UTC m=+2137.650667254"
Sep 07 11:58:13.938091 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:13.938058    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75d79b99ff-xg24v"
Sep 07 11:58:22.534964 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:22.534932    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"]
Sep 07 11:58:22.535516 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:22.535148    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9" podUID="529dc0e4-9e4e-4294-a202-2d292fae5c5f" containerName="maas-api" containerID="cri-o://ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05" gracePeriod=30
Sep 07 11:58:23.876050 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:23.876023    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:58:24.002084 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.002049    2581 generic.go:356] "Generic (PLEG): container finished" podID="529dc0e4-9e4e-4294-a202-2d292fae5c5f" containerID="ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05" exitCode=0
Sep 07 11:58:24.002251 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.002100    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"
Sep 07 11:58:24.002251 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.002103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9" event={"ID":"529dc0e4-9e4e-4294-a202-2d292fae5c5f","Type":"ContainerDied","Data":"ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05"}
Sep 07 11:58:24.002251 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.002215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9" event={"ID":"529dc0e4-9e4e-4294-a202-2d292fae5c5f","Type":"ContainerDied","Data":"1fe70ae8fc813f2cde5c8a37c3f5080c47b4f0f63cb1c87fbf99e3ad951dc6e9"}
Sep 07 11:58:24.002251 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.002236    2581 scope.go:117] "RemoveContainer" containerID="ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05"
Sep 07 11:58:24.011033 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.011012    2581 scope.go:117] "RemoveContainer" containerID="ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05"
Sep 07 11:58:24.011194 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.011177    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529dc0e4-9e4e-4294-a202-2d292fae5c5f-openshift-service-ca\") pod \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") "
Sep 07 11:58:24.011278 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.011263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-maas-api-tls\") pod \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") "
Sep 07 11:58:24.011327 ip-10-0-143-241 kubenswrapper[2581]: E0907 11:58:24.011279    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05\": container with ID starting with ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05 not found: ID does not exist" containerID="ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05"
Sep 07 11:58:24.011327 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.011309    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05"} err="failed to get container status \"ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05\": rpc error: code = NotFound desc = could not find container \"ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05\": container with ID starting with ac4706518492f8ea0670f4cf3e184ef9b6cf3f29b6c8b1244798af15c084ff05 not found: ID does not exist"
Sep 07 11:58:24.011448 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.011327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4gbh\" (UniqueName: \"kubernetes.io/projected/529dc0e4-9e4e-4294-a202-2d292fae5c5f-kube-api-access-s4gbh\") pod \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") "
Sep 07 11:58:24.011448 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.011358    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-metrics-certs\") pod \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\" (UID: \"529dc0e4-9e4e-4294-a202-2d292fae5c5f\") "
Sep 07 11:58:24.011559 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.011508    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529dc0e4-9e4e-4294-a202-2d292fae5c5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "529dc0e4-9e4e-4294-a202-2d292fae5c5f" (UID: "529dc0e4-9e4e-4294-a202-2d292fae5c5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 11:58:24.011643 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.011615    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529dc0e4-9e4e-4294-a202-2d292fae5c5f-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:58:24.013394 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.013366    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "529dc0e4-9e4e-4294-a202-2d292fae5c5f" (UID: "529dc0e4-9e4e-4294-a202-2d292fae5c5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:58:24.013513 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.013460    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "529dc0e4-9e4e-4294-a202-2d292fae5c5f" (UID: "529dc0e4-9e4e-4294-a202-2d292fae5c5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:58:24.013513 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.013478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529dc0e4-9e4e-4294-a202-2d292fae5c5f-kube-api-access-s4gbh" (OuterVolumeSpecName: "kube-api-access-s4gbh") pod "529dc0e4-9e4e-4294-a202-2d292fae5c5f" (UID: "529dc0e4-9e4e-4294-a202-2d292fae5c5f"). InnerVolumeSpecName "kube-api-access-s4gbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:58:24.113078 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.113041    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:58:24.113078 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.113074    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4gbh\" (UniqueName: \"kubernetes.io/projected/529dc0e4-9e4e-4294-a202-2d292fae5c5f-kube-api-access-s4gbh\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:58:24.113078 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.113086    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529dc0e4-9e4e-4294-a202-2d292fae5c5f-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 07 11:58:24.325336 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.325304    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"]
Sep 07 11:58:24.330585 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.330557    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f7b4565f-dswq9"]
Sep 07 11:58:24.940919 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:24.940882    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="529dc0e4-9e4e-4294-a202-2d292fae5c5f" path="/var/lib/kubelet/pods/529dc0e4-9e4e-4294-a202-2d292fae5c5f/volumes"
Sep 07 11:58:44.954082 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:44.954053    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pdwnc_7c6de5af-c4ca-4006-b860-8269604c34c7/manager/0.log"
Sep 07 11:58:45.073823 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:45.073779    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dg9rw_64798131-40bb-44d6-8461-808df5e5e9cf/manager/0.log"
Sep 07 11:58:45.566253 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:45.566220    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f479d7474-bz4hv_a76fc137-c23c-4d98-9b77-a769f24eafde/copy-manifests/0.log"
Sep 07 11:58:45.572734 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:45.572708    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f479d7474-bz4hv_a76fc137-c23c-4d98-9b77-a769f24eafde/manager/0.log"
Sep 07 11:58:46.318286 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:46.318252    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-75d79b99ff-xg24v_e516238d-eb33-4138-b6b1-b2075096b1e5/maas-api/0.log"
Sep 07 11:58:47.529212 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:47.529181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-r5c8l_87974954-50b8-4fe4-95fa-17dd6c1f2c39/manager/0.log"
Sep 07 11:58:47.906837 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:47.906743    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-g7xhf_77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb/manager/0.log"
Sep 07 11:58:48.689496 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:48.689463    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nrvfw_3258b698-e714-4e3f-a3be-09039586bd22/discovery/0.log"
Sep 07 11:58:48.926311 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:48.926279    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586fdf7b-tz4vp_06e006b6-b789-4fc3-a699-1406354e6c58/kube-auth-proxy/0.log"
Sep 07 11:58:49.445726 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:49.445694    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5895868fb6-s4mnj_ffc78f96-d803-4e3e-bf08-a96f5b7e2aab/router/0.log"
Sep 07 11:58:49.927632 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:49.927587    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-8l9r2_7566aea8-4484-418c-9887-486f12ac16ec/main/0.log"
Sep 07 11:58:50.284705 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:50.284673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkjwvt_6675f584-648a-4b6d-b281-566783d4beb2/main/0.log"
Sep 07 11:58:50.525218 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:50.525188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zqhwp_237d0dd9-78bd-4d5c-9d49-ffdc30bda092/main/0.log"
Sep 07 11:58:59.888672 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:58:59.888638    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nkt94_3d0f4eb6-2bba-480e-8dae-1827c5ac731a/global-pull-secret-syncer/0.log"
Sep 07 11:59:00.036121 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:00.036091    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z4xbr_40b2c47e-d219-4889-b4ed-d78eaa2f179e/konnectivity-agent/0.log"
Sep 07 11:59:00.096580 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:00.096548    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-241.ec2.internal_20ada389393df0af1b24f49af160350a/haproxy/0.log"
Sep 07 11:59:04.566727 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:04.566693    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-r5c8l_87974954-50b8-4fe4-95fa-17dd6c1f2c39/manager/0.log"
Sep 07 11:59:04.648249 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:04.648222    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-g7xhf_77fbec0c-5f4e-434f-8fa3-a4e0d641f9cb/manager/0.log"
Sep 07 11:59:06.180218 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.180183    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37/alertmanager/0.log"
Sep 07 11:59:06.200387 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.200366    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37/config-reloader/0.log"
Sep 07 11:59:06.218932 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.218905    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37/kube-rbac-proxy-web/0.log"
Sep 07 11:59:06.240203 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.240181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37/kube-rbac-proxy/0.log"
Sep 07 11:59:06.258081 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.258059    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37/kube-rbac-proxy-metric/0.log"
Sep 07 11:59:06.277308 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.277287    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37/prom-label-proxy/0.log"
Sep 07 11:59:06.295457 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.295439    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba2efefd-a8a5-4afe-ae0e-40a7f0fc3a37/init-config-reloader/0.log"
Sep 07 11:59:06.458592 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.458522    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-spjrj_5b1f75c4-57e6-4e6b-98fa-9640c4a6ce83/monitoring-plugin/0.log"
Sep 07 11:59:06.556036 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.556007    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh8c5_aadab419-cf81-41df-8c04-d0a8c224205e/node-exporter/0.log"
Sep 07 11:59:06.583539 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.583517    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh8c5_aadab419-cf81-41df-8c04-d0a8c224205e/kube-rbac-proxy/0.log"
Sep 07 11:59:06.605228 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:06.605207    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh8c5_aadab419-cf81-41df-8c04-d0a8c224205e/init-textfile/0.log"
Sep 07 11:59:07.208046 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:07.208019    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d74f69f95-txmmm_61a57663-07ba-4c20-a30f-7b0bae9f52e5/thanos-query/0.log"
Sep 07 11:59:07.226425 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:07.226403    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d74f69f95-txmmm_61a57663-07ba-4c20-a30f-7b0bae9f52e5/kube-rbac-proxy-web/0.log"
Sep 07 11:59:07.248625 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:07.248602    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d74f69f95-txmmm_61a57663-07ba-4c20-a30f-7b0bae9f52e5/kube-rbac-proxy/0.log"
Sep 07 11:59:07.267275 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:07.267218    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d74f69f95-txmmm_61a57663-07ba-4c20-a30f-7b0bae9f52e5/prom-label-proxy/0.log"
Sep 07 11:59:07.285835 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:07.285784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d74f69f95-txmmm_61a57663-07ba-4c20-a30f-7b0bae9f52e5/kube-rbac-proxy-rules/0.log"
Sep 07 11:59:07.304910 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:07.304889    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d74f69f95-txmmm_61a57663-07ba-4c20-a30f-7b0bae9f52e5/kube-rbac-proxy-metrics/0.log"
Sep 07 11:59:09.852803 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.852758    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"]
Sep 07 11:59:09.855140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.853154    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="529dc0e4-9e4e-4294-a202-2d292fae5c5f" containerName="maas-api"
Sep 07 11:59:09.855140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.853168    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="529dc0e4-9e4e-4294-a202-2d292fae5c5f" containerName="maas-api"
Sep 07 11:59:09.855140 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.853220    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="529dc0e4-9e4e-4294-a202-2d292fae5c5f" containerName="maas-api"
Sep 07 11:59:09.856043 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.856028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:09.858536 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.858515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f2qg\"/\"kube-root-ca.crt\""
Sep 07 11:59:09.858536 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.858528    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f2qg\"/\"openshift-service-ca.crt\""
Sep 07 11:59:09.858695 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.858516    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9f2qg\"/\"default-dockercfg-svqxd\""
Sep 07 11:59:09.864893 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:09.864872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"]
Sep 07 11:59:10.007760 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.007719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-lib-modules\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.007983 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.007865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg27z\" (UniqueName: \"kubernetes.io/projected/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-kube-api-access-qg27z\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.008053 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.008002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-proc\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.008053 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.008028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-podres\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.008168 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.008076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-sys\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg27z\" (UniqueName: \"kubernetes.io/projected/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-kube-api-access-qg27z\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-proc\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109369 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-podres\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109602 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-sys\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109602 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-proc\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109602 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-lib-modules\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109602 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109513    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-sys\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109602 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-lib-modules\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.109602 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.109546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-podres\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.118100 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.118080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg27z\" (UniqueName: \"kubernetes.io/projected/dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1-kube-api-access-qg27z\") pod \"perf-node-gather-daemonset-stgzv\" (UID: \"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1\") " pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.165961 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.165936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:10.434194 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:10.434166    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"]
Sep 07 11:59:10.436012 ip-10-0-143-241 kubenswrapper[2581]: W0907 11:59:10.435984    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddfdbd2d7_26ef_4359_bdc1_a9a20cbeeaa1.slice/crio-baa2989a9e8ba093b7e4a54fa308a7faf4bba19406c402ac4b035553b992b53c WatchSource:0}: Error finding container baa2989a9e8ba093b7e4a54fa308a7faf4bba19406c402ac4b035553b992b53c: Status 404 returned error can't find the container with id baa2989a9e8ba093b7e4a54fa308a7faf4bba19406c402ac4b035553b992b53c
Sep 07 11:59:11.223673 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:11.223636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv" event={"ID":"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1","Type":"ContainerStarted","Data":"25cb50b14d0030289009296e3e924880ed62ff95e981caec9fa4b918de364636"}
Sep 07 11:59:11.224178 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:11.223688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv" event={"ID":"dfdbd2d7-26ef-4359-bdc1-a9a20cbeeaa1","Type":"ContainerStarted","Data":"baa2989a9e8ba093b7e4a54fa308a7faf4bba19406c402ac4b035553b992b53c"}
Sep 07 11:59:11.224178 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:11.223809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:11.238911 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:11.238868    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv" podStartSLOduration=2.23885461 podStartE2EDuration="2.23885461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:59:11.237890599 +0000 UTC m=+2200.943671288" watchObservedRunningTime="2026-09-07 11:59:11.23885461 +0000 UTC m=+2200.944635299"
Sep 07 11:59:11.868581 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:11.868546    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h9qtv_8aecb463-b248-47aa-8ac7-0e15b220c491/dns/0.log"
Sep 07 11:59:11.886307 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:11.886279    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h9qtv_8aecb463-b248-47aa-8ac7-0e15b220c491/kube-rbac-proxy/0.log"
Sep 07 11:59:12.027076 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:12.027047    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m8j27_dce8b7bd-5f71-4d33-a6f5-dfe8ae3c02b1/dns-node-resolver/0.log"
Sep 07 11:59:12.552717 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:12.552684    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lt54v_52a43c8b-4eaf-4514-a60f-d0fba1696bd0/node-ca/0.log"
Sep 07 11:59:13.852431 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:13.852400    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nrvfw_3258b698-e714-4e3f-a3be-09039586bd22/discovery/0.log"
Sep 07 11:59:13.895519 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:13.895493    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586fdf7b-tz4vp_06e006b6-b789-4fc3-a699-1406354e6c58/kube-auth-proxy/0.log"
Sep 07 11:59:14.017331 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:14.017299    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5895868fb6-s4mnj_ffc78f96-d803-4e3e-bf08-a96f5b7e2aab/router/0.log"
Sep 07 11:59:14.857842 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:14.857815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5crfb_a74ec153-c250-4823-9bef-93811a97c81d/serve-healthcheck-canary/0.log"
Sep 07 11:59:15.481030 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:15.480994    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhkmc_93118025-9b9a-4565-ab00-2de67090cf52/kube-rbac-proxy/0.log"
Sep 07 11:59:15.497629 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:15.497600    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhkmc_93118025-9b9a-4565-ab00-2de67090cf52/exporter/0.log"
Sep 07 11:59:15.514548 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:15.514525    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhkmc_93118025-9b9a-4565-ab00-2de67090cf52/extractor/0.log"
Sep 07 11:59:17.238735 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:17.238710    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9f2qg/perf-node-gather-daemonset-stgzv"
Sep 07 11:59:17.557676 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:17.557599    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pdwnc_7c6de5af-c4ca-4006-b860-8269604c34c7/manager/0.log"
Sep 07 11:59:17.726912 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:17.726877    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dg9rw_64798131-40bb-44d6-8461-808df5e5e9cf/manager/0.log"
Sep 07 11:59:17.920761 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:17.920682    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f479d7474-bz4hv_a76fc137-c23c-4d98-9b77-a769f24eafde/manager/0.log"
Sep 07 11:59:17.939623 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:17.939597    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f479d7474-bz4hv_a76fc137-c23c-4d98-9b77-a769f24eafde/copy-manifests/0.log"
Sep 07 11:59:19.535860 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:19.535828    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jwkst_d2d38264-0d09-4638-8779-917c2600ac79/openshift-lws-operator/0.log"
Sep 07 11:59:20.391200 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:20.391169    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nrvfw_3258b698-e714-4e3f-a3be-09039586bd22/discovery/0.log"
Sep 07 11:59:20.413523 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:20.413496    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586fdf7b-tz4vp_06e006b6-b789-4fc3-a699-1406354e6c58/kube-auth-proxy/0.log"
Sep 07 11:59:20.510364 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:20.510333    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5895868fb6-s4mnj_ffc78f96-d803-4e3e-bf08-a96f5b7e2aab/router/0.log"
Sep 07 11:59:22.315533 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:22.315504    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nrvfw_3258b698-e714-4e3f-a3be-09039586bd22/discovery/0.log"
Sep 07 11:59:22.354862 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:22.354826    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586fdf7b-tz4vp_06e006b6-b789-4fc3-a699-1406354e6c58/kube-auth-proxy/0.log"
Sep 07 11:59:22.450509 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:22.450473    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5895868fb6-s4mnj_ffc78f96-d803-4e3e-bf08-a96f5b7e2aab/router/0.log"
Sep 07 11:59:25.901113 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:25.901030    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9hpnc_b1a3c173-d443-479b-9521-edf01ab367e8/kube-multus-additional-cni-plugins/0.log"
Sep 07 11:59:25.919027 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:25.919005    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9hpnc_b1a3c173-d443-479b-9521-edf01ab367e8/egress-router-binary-copy/0.log"
Sep 07 11:59:25.936295 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:25.936271    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9hpnc_b1a3c173-d443-479b-9521-edf01ab367e8/cni-plugins/0.log"
Sep 07 11:59:25.956080 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:25.956058    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9hpnc_b1a3c173-d443-479b-9521-edf01ab367e8/bond-cni-plugin/0.log"
Sep 07 11:59:25.973893 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:25.973861    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9hpnc_b1a3c173-d443-479b-9521-edf01ab367e8/routeoverride-cni/0.log"
Sep 07 11:59:25.990770 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:25.990748    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9hpnc_b1a3c173-d443-479b-9521-edf01ab367e8/whereabouts-cni-bincopy/0.log"
Sep 07 11:59:26.009511 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:26.009484    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9hpnc_b1a3c173-d443-479b-9521-edf01ab367e8/whereabouts-cni/0.log"
Sep 07 11:59:26.403883 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:26.403847    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xpwmp_43925274-591e-4a1c-972f-aba80b98abbf/kube-multus/0.log"
Sep 07 11:59:26.467249 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:26.467217    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lmtxp_85e5bbff-0cc8-45cb-91b8-b106b0a5a978/network-metrics-daemon/0.log"
Sep 07 11:59:26.484613 ip-10-0-143-241 kubenswrapper[2581]: I0907 11:59:26.484589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lmtxp_85e5bbff-0cc8-45cb-91b8-b106b0a5a978/kube-rbac-proxy/0.log"